<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    Hi Shwetha, <br>
    <br>
    I ran the script against against the first brick in my cluster
    (after shutting gluster down) and ran into a small issue: <br>
    <font face="monospace"><br>
      [root@storage01 ~]# ./glusterfs-georep-upgrade.py
      /data/storage_a/storage/ 2&gt;&amp;1 | tee -a
      /var/log/storage_a.txt<br>
      Traceback (most recent call last):<br>
        File "./glusterfs-georep-upgrade.py", line 77, in &lt;module&gt;<br>
          modify_htime_file(args.brick_path)<br>
        File "./glusterfs-georep-upgrade.py", line 61, in
      modify_htime_file<br>
          shutil.copyfile(pth, os.path.join(path, changelog))<br>
        File "/usr/lib64/python3.6/shutil.py", line 104, in copyfile<br>
          raise SameFileError("{!r} and {!r} are the same
      file".format(src, dst))<br>
      shutil.SameFileError:
'/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/CHANGELOG.1615318474'
      and
'/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/CHANGELOG.1615318474'
      are the same file</font><br>
    <br>
    Since this script wasn't run prior to the upgrade some of the files
    in the original HTIME file are already in the year/month/date format
    - so the copy operation fails on those files terminating the loop
    early, meaning that not all of the files make it into the
    temp_htime_file, and the new htime file doesn't get put into place.
    <br>
    <font face="monospace"><br>
      [root@storage01 changelogs]# tail -c 256 htime/HTIME.1597342860 <br>
/changelog.1616431368/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431382/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431396/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431410[root@storage01
      changelogs]# <br>
      [root@storage01 changelogs]# <br>
      [root@storage01 changelogs]# tail -c 256 htime/temp_htime_file <br>
/changelog.1615317943/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/changelog.1615317957/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/changelog.1615317972/data/storage_a/storage/.glusterfs/changelogs/2021/03/09/CHANGELOG.1615318474[root@storage01
      changelogs]#<br>
      <br>
      [root@storage01 changelogs]# ls -lh<br>
      total 4.0K<br>
      drw-------. 7 root root 80 Mar 22 10:07 2020<br>
      drw-------. 5 root root 52 Mar 22 10:07 2021<br>
      -rw-r--r--. 1 root root 51 Mar 22 09:43 CHANGELOG<br>
      drw-------. 2 root root 10 May  7  2020 csnap<br>
      drw-------. 2 root root 65 Mar 22 09:57 htime<br>
      [root@storage01 changelogs]# ls -lh 2021/03/<br>
      total 24K<br>
      drw-------. 2 root root 16K Mar 22 10:07 01<br>
      drw-------. 2 root root  74 Mar  9 12:19 09<br>
      drw-------. 2 root root  42 Mar 15 14:08 15<br>
      drw-------. 2 root root  42 Mar 16 12:04 16<br>
      [root@storage01 changelogs]# ls -lh htime/<br>
      total 178M<br>
      -rw-r--r--. 1 root root 86M Mar 22 09:43 HTIME.1597342860<br>
      -rw-r--r--. 1 root root 93M Mar 22 10:07 temp_htime_file<br>
       </font><br>
    <br>
    I assume here I can just manually add the remaining changelog files
    into the temp_htime_file and then perform the backup and
    replacement, but wanted to double check. <br>
    <br>
    It's also strange that there are files listed in the original
    HTIME.1597342860 file which don't exist on the filesystem.... the
    directories end at 2021/03/16, but there are files listed in the
    2021/03/22 . Please advise. <br>
    <br>
    <font face="monospace">[root@storage01 changelogs]# find 2021/03/
      -type f | egrep -v "^2021/03/01"<br>
      2021/03/09/CHANGELOG.1615318474<br>
      2021/03/09/CHANGELOG.1615321197<br>
      2021/03/15/CHANGELOG.1615842490<br>
      2021/03/16/CHANGELOG.1615921482<br>
      <br>
      [root@storage01 changelogs]# tail -c 2048 htime/HTIME.1597342860 <br>
hangelog.1616431028/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431043/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431058/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431073/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431088/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431103/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431118/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431133/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431148/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431163/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431178/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431193/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431208/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431223/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431238/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431253/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431268/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431283/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431298/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431312/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431326/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431340/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431354/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431368/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431382/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431396/data/storage_a/storage/.glusterfs/changelogs/2021/03/22/changelog.1616431410</font><br>
    <br>
    <br>
    Thanks,<br>
     -Matthew<br>
    <div class="moz-signature"><font size="-1">
        <p>--<br>
          <br>
        </p>
      </font>
    </div>
    <div class="moz-cite-prefix">On 3/19/21 5:36 AM, Shwetha Acharya
      wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAERh03qtOiZg_hLAZtGjZ0Y6QbK-4R38HTf_=hs4iiUNzotCyQ@mail.gmail.com">
      <div>
        Notice: This message was sent from outside the University of
        Victoria email system. Please be cautious with links and
        sensitive information.
      </div>
      <br>
      <div>
        <div dir="ltr"> <a class="gmail_plusreply" id="plusReplyChip-1"
            href="mailto:kaleb@redhat.com" tabindex="-1"
            moz-do-not-send="true">@Kaleb Keithley</a> <a
            class="gmail_plusreply" id="plusReplyChip-2"
            href="mailto:khiremat@redhat.com" tabindex="-1"
            moz-do-not-send="true">@Kotresh Hiremath Ravishankar</a> 
          any insights on use of rpm scriplets for this script?
          Basically this script is to be run before the upgradation.<br>
          <br>
          Strahil,<br>
          I will update the docs once we decide with respect to rpm
          scriptlet part.<br>
          <br>
          Regrads,<br>
          Shwetha  <br>
          <br>
        </div>
        <br>
        <div class="gmail_quote">
          <div dir="ltr" class="gmail_attr">On Fri, Mar 19, 2021 at 5:45
            PM Strahil Nikolov &lt;<a
              href="mailto:hunter86_bg@yahoo.com" moz-do-not-send="true">hunter86_bg@yahoo.com</a>&gt;
            wrote:<br>
          </div>
          <blockquote class="gmail_quote">
            Hi Shwetha,
            <div><br>
            </div>
            <div
              id="gmail-m_634331480037335435yMail_cursorElementTracker_1616155936663">Is
              that script mentioned in the documentation (release notes
              for example) ?</div>
            <div
              id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156103839"><br>
            </div>
            <div
              id="gmail-m_634331480037335435yMail_cursorElementTracker_1616155936803">Do
              you think that the new glusterfs rpm can call (via rpm
              scriplets) the script safely ?</div>
            <div
              id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156033372"><br>
            </div>
            <div
              id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156004182"><br>
            </div>
            <div
              id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156004390">If
              yes, then it will be nice if the rpm itself runs the
              script instead of the user.</div>
            <div
              id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156004390"><br>
            </div>
            <div
              id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156077707"><br>
            </div>
            <div
              id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156077811">Best
              Regards,</div>
            <div
              id="gmail-m_634331480037335435yMail_cursorElementTracker_1616156082780">Strahil
              Nikolov<br>
              <br>
              <blockquote>
                <div>
                  <div>On Fri, Mar 19, 2021 at 11:44, Shwetha Acharya</div>
                  <div>&lt;<a href="mailto:sacharya@redhat.com"
                      target="_blank" moz-do-not-send="true">sacharya@redhat.com</a>&gt;
                    wrote:</div>
                </div>
                <div>
                  <div id="gmail-m_634331480037335435yiv1091814412">
                    <div dir="ltr">Hi Mathew,<br>
                      <br>
                      The upgrade script <a rel="nofollow noopener
                        noreferrer"
href="https://github.com/gluster/glusterfs/commit/2857fe3fad4d2b30894847088a54b847b88a23b9"
                        target="_blank" moz-do-not-send="true">https://github.com/gluster/glusterfs/commit/2857fe3fad4d2b30894847088a54b847b88a23b9</a>
                      need to be run, to make sure that the changelogs
                      and htimes files created in the older version are
                      updated as per the new changes in the directory
                      structure.<br>
                      <br>
                      If not done, search algorithm during history crawl
                      would give wrong result.<br>
                      <br>
                      Regards,<br>
                      Shwetha</div>
                    <br>
                    <div>
                      <div dir="ltr">On Fri, Mar 19, 2021 at 3:53 AM
                        Strahil Nikolov &lt;<a rel="nofollow noopener
                          noreferrer"
                          href="mailto:hunter86_bg@yahoo.com"
                          target="_blank" moz-do-not-send="true">hunter86_bg@yahoo.com</a>&gt;
                        wrote:<br>
                      </div>
                      <blockquote>
                        Sadly,
                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106048432"><br>
                        </div>
                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106048546">I'm
                          out of ideas. It makes sense... if the
                          changelog was changed - then it won't work
                          after the upgrade.</div>
                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106048546"><br>
                        </div>
                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106093562">I
                          guess that once you delete the session, you
                          can remove the extended attribute for the
                          time.</div>
                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106136564"><br>
                        </div>
                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106136756">Best
                          Regards,</div>
                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106141079">Strahil
                          Nikolov</div>
                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yMail_cursorElementTracker_1616106094445"><br>
                          <blockquote>
                            <div>
                              <div>On Wed, Mar 17, 2021 at 23:11,
                                Matthew Benstead</div>
                              <div>&lt;<a rel="nofollow noopener
                                  noreferrer"
                                  href="mailto:matthewb@uvic.ca"
                                  target="_blank" moz-do-not-send="true">matthewb@uvic.ca</a>&gt;
                                wrote:</div>
                            </div>
                            <div>
                              <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
                                <div>Yes, I've run through everything
                                  short of regenerating the keys and
                                  creating the session again with no
                                  errors. Everything looks ok.
                                  <br>
                                  <br>
                                  But I did notice that the changelog
                                  format had changed, instead of them
                                  being dumped into one directory, they
                                  now seem to be separated in
                                  year/month/day directories...
                                  <br>
                                  <br>
                                  Looks like this change in 8.0: <a
                                    rel="nofollow noopener noreferrer"
                                    shape="rect"
                                    href="https://github.com/gluster/glusterfs/issues/154"
                                    target="_blank"
                                    moz-do-not-send="true">
https://github.com/gluster/glusterfs/issues/154</a><br>
                                  <br>
                                  [root@storage01 changelogs]# ls -lh |
                                  head<br>
                                  total 16G<br>
                                  drw-------. 3 root root   24 Mar  9
                                  11:34 2021<br>
                                  -rw-r--r--. 1 root root   51 Mar 17
                                  13:19 CHANGELOG<br>
                                  -rw-r--r--. 1 root root  13K Aug 13 
                                  2020 CHANGELOG.1597343197<br>
                                  -rw-r--r--. 1 root root  51K Aug 13 
                                  2020 CHANGELOG.1597343212<br>
                                  -rw-r--r--. 1 root root  86K Aug 13 
                                  2020 CHANGELOG.1597343227<br>
                                  -rw-r--r--. 1 root root  99K Aug 13 
                                  2020 CHANGELOG.1597343242<br>
                                  -rw-r--r--. 1 root root  69K Aug 13 
                                  2020 CHANGELOG.1597343257<br>
                                  -rw-r--r--. 1 root root  69K Aug 13 
                                  2020 CHANGELOG.1597343272<br>
                                  -rw-r--r--. 1 root root  72K Aug 13 
                                  2020 CHANGELOG.1597343287<br>
                                  [root@storage01 changelogs]# ls -lh |
                                  tail<br>
                                  -rw-r--r--. 1 root root   92 Mar  1
                                  21:33 CHANGELOG.1614663193<br>
                                  -rw-r--r--. 1 root root   92 Mar  1
                                  21:42 CHANGELOG.1614663731<br>
                                  -rw-r--r--. 1 root root   92 Mar  1
                                  21:42 CHANGELOG.1614663760<br>
                                  -rw-r--r--. 1 root root  511 Mar  1
                                  21:47 CHANGELOG.1614664043<br>
                                  -rw-r--r--. 1 root root  536 Mar  1
                                  21:48 CHANGELOG.1614664101<br>
                                  -rw-r--r--. 1 root root 2.8K Mar  1
                                  21:48 CHANGELOG.1614664116<br>
                                  -rw-r--r--. 1 root root   92 Mar  1
                                  22:20 CHANGELOG.1614666061<br>
                                  -rw-r--r--. 1 root root   92 Mar  1
                                  22:29 CHANGELOG.1614666554<br>
                                  drw-------. 2 root root   10 May  7 
                                  2020 csnap<br>
                                  drw-------. 2 root root   38 Aug 13 
                                  2020 htime<br>
                                  <br>
                                  <br>
                                  [root@storage01 changelogs]# ls -lh
                                  2021/03/09/<br>
                                  total 8.0K<br>
                                  -rw-r--r--. 1 root root 51 Mar  9
                                  11:26 CHANGELOG.1615318474<br>
                                  -rw-r--r--. 1 root root 51 Mar  9
                                  12:19 CHANGELOG.1615321197<br>
                                  [root@storage01 changelogs]# ls -lh
                                  2021/03/15/<br>
                                  total 4.0K<br>
                                  -rw-r--r--. 1 root root 51 Mar 15
                                  13:38 CHANGELOG.1615842490<br>
                                  [root@storage01 changelogs]# ls -lh
                                  2021/03/16<br>
                                  total 4.0K<br>
                                  -rw-r--r--. 1 root root 331 Mar 16
                                  12:04 CHANGELOG.1615921482<br>
                                  <br>
                                  But it looks like the htime file still
                                  records them...<br>
                                  <br>
                                  [root@storage01 changelogs]# ls -lh
                                  htime<br>
                                  total 84M<br>
                                  -rw-r--r--. 1 root root 84M Mar 17
                                  13:31 HTIME.1597342860<br>
                                  <br>
                                  [root@storage01 changelogs]# head -c
                                  256 htime/HTIME.1597342860 <br>
/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342875/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342890/data/storage_a/storage/.glusterfs/changelogs/changelog.1597342904/data/storage_a/storage/.glusterfs/changelogs/changelog[root@storage01
                                  changelogs]# <br>
                                  <br>
                                  [root@storage01 changelogs]# tail -c
                                  256 htime/HTIME.1597342860 <br>
/changelog.1616013484/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013499/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013514/data/storage_a/storage/.glusterfs/changelogs/2021/03/17/changelog.1616013529[root@storage01
                                  changelogs]# <br>
                                  <br>
                                  <br>
                                  And there seems to be an xattr for
                                  time in the brick root - presumably
                                  for when changelogs were enabled:
                                  <br>
                                  <br>
                                  [root@storage01 changelogs]# getfattr
                                  -d -m. -e hex /data/storage_a/storage
                                  2&gt;&amp;1 | egrep xtime<br>
trusted.glusterfs.cf94a8f2-324b-40b3-bf72-c3766100ea99.xtime=0x60510140000ef317<br>
                                  <br>
                                  <br>
                                  Reading through the changes, it looks
                                  like there is a script to convert from
                                  one format to the other... I didn't
                                  see anything in the release notes for
                                  8.0 about it... this seems like it
                                  could be the fix, and explain why
                                  gluster can't get through the
                                  changelogs....  Thoughts? <br>
                                  <ul>
                                    <li><a rel="nofollow noopener
                                        noreferrer" shape="rect"
href="https://github.com/gluster/glusterfs/issues/154#issuecomment-585701964"
                                        target="_blank"
                                        moz-do-not-send="true">https://github.com/gluster/glusterfs/issues/154#issuecomment-585701964</a></li>
                                    <li><a rel="nofollow noopener
                                        noreferrer" shape="rect"
                                        href="https://review.gluster.org/#/c/glusterfs/+/24121/"
                                        target="_blank"
                                        moz-do-not-send="true">https://review.gluster.org/#/c/glusterfs/+/24121/</a><br>
                                    </li>
                                  </ul>
                                  <br>
                                  Thanks,<br>
                                   -Matthew<br>
                                  <div>
                                    <p>--<br>
                                      Matthew Benstead<br>
                                      System Administrator<br>
                                      <a rel="nofollow noopener
                                        noreferrer" shape="rect"
                                        href="https://pacificclimate.org/"
                                        target="_blank"
                                        moz-do-not-send="true">Pacific
                                        Climate Impacts Consortium</a><br>
                                      University of Victoria, UH1<br>
                                      PO Box 1800, STN CSC<br>
                                      Victoria, BC, V8W 2Y2<br>
                                      Phone: +1-250-721-8432<br>
                                      Email: <a rel="nofollow noopener
                                        noreferrer" shape="rect"
                                        href="mailto:matthewb@uvic.ca"
                                        target="_blank"
                                        moz-do-not-send="true">
                                        matthewb@uvic.ca</a></p>
                                  </div>
                                  <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yqt01812">
                                    <div>On 3/16/21 9:36 PM, Strahil
                                      Nikolov wrote:<br>
                                    </div>
                                    <blockquote type="cite">
                                      <div>Notice: This message was sent
                                        from outside the University of
                                        Victoria email system. Please be
                                        cautious with links and
                                        sensitive information.
                                      </div>
                                      <br>
                                      <div>Have you verified all steps
                                        for creating the geo-replication
                                        ?
                                        <div><br>
                                        </div>
                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955616619">If
                                          yes , maybe using
                                          "reset-sync-time + delete +
                                          create" makes sense.Keep in
                                          mind that it will take a long
                                          time once the geo-rep is
                                          established again.</div>
                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955729476"><br>
                                        </div>
                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955750689"><br>
                                        </div>
                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955750840">Best
                                          Regards,</div>
                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955755206">Strahil
                                          Nikolov</div>
                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615955730266"><br>
                                          <blockquote>
                                            <div>
                                              <div>On Tue, Mar 16, 2021
                                                at 22:34, Matthew
                                                Benstead</div>
                                              <div><a rel="nofollow
                                                  noopener noreferrer"
                                                  shape="rect"
                                                  href="mailto:matthewb@uvic.ca"
                                                  target="_blank"
                                                  moz-do-not-send="true">&lt;matthewb@uvic.ca&gt;</a>
                                                wrote:</div>
                                            </div>
                                            <div>
                                              <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
                                                <div>Thanks Strahil, <br>
                                                  <br>
                                                  I wanted to make sure
                                                  the issue wasn't
                                                  occurring because
                                                  there were no new
                                                  changes to sync from
                                                  the master volume. So
                                                  I created some files
                                                  and restarted the
                                                  sync, but it had no
                                                  effect.
                                                  <br>
                                                  <br>
                                                  [root@storage01 ~]# cd
                                                  /storage2/home/test/<br>
                                                  [root@storage01 test]#
                                                  for nums in
                                                  {1,2,3,4,5,6,7,8,9,0};
                                                  do touch $nums.txt;
                                                  done<br>
                                                  <br>
                                                  [root@storage01 test]#
                                                  gluster volume
                                                  geo-replication
                                                  storage <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> start<br>
                                                  Starting
                                                  geo-replication
                                                  session between
                                                  storage &amp; <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> has been successful<br>
                                                  [root@storage01 test]#
                                                  gluster volume
                                                  geo-replication status
                                                  <br>
                                                   <br>
                                                  MASTER NODE    MASTER
                                                  VOL    MASTER
                                                  BRICK              
                                                  SLAVE USER   
                                                  SLAVE                                       
                                                  SLAVE NODE   
                                                  STATUS            
                                                  CRAWL STATUS   
                                                  LAST_SYNCED         
                                                  <br>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
                                                  10.0.231.91   
                                                  storage      
                                                  /data/storage_a/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                  Initializing...   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.91   
                                                  storage      
                                                  /data/storage_c/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                  Initializing...   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.91   
                                                  storage      
                                                  /data/storage_b/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                  Initializing...   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.93   
                                                  storage      
                                                  /data/storage_c/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                  Initializing...   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.93   
                                                  storage      
                                                  /data/storage_b/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                  Initializing...   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.93   
                                                  storage      
                                                  /data/storage_a/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                  Initializing...   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.92   
                                                  storage      
                                                  /data/storage_b/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                  Initializing...   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.92   
                                                  storage      
                                                  /data/storage_a/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                  Initializing...   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.92   
                                                  storage      
                                                  /data/storage_c/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                  Initializing...   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  [root@storage01 test]#
                                                  gluster volume
                                                  geo-replication status
                                                  <br>
                                                   <br>
                                                  MASTER NODE    MASTER
                                                  VOL    MASTER
                                                  BRICK              
                                                  SLAVE USER   
                                                  SLAVE                                       
                                                  SLAVE NODE   
                                                  STATUS    CRAWL
                                                  STATUS   
                                                  LAST_SYNCED         
                                                  <br>
---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
                                                  10.0.231.91   
                                                  storage      
                                                  /data/storage_a/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.91   
                                                  storage      
                                                  /data/storage_c/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.91   
                                                  storage      
                                                  /data/storage_b/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.93   
                                                  storage      
                                                  /data/storage_c/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.93   
                                                  storage      
                                                  /data/storage_b/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.93   
                                                  storage      
                                                  /data/storage_a/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.92   
                                                  storage      
                                                  /data/storage_b/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.92   
                                                  storage      
                                                  /data/storage_a/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  10.0.231.92   
                                                  storage      
                                                  /data/storage_c/storage   
                                                  geoaccount    <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
                                                  N/A            
                                                  N/A                 
                                                  <br>
                                                  [root@storage01 test]#
                                                  gluster volume
                                                  geo-replication
                                                  storage <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> stop<br>
                                                  Stopping
                                                  geo-replication
                                                  session between
                                                  storage &amp; <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:geoaccount@10.0.231.81::pcic-backup"
                                                    target="_blank"
                                                    moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> has been successful<br>
                                                  <br>
                                                  Still getting the same
                                                  error about the
                                                  history crawl failing:
                                                  <br>
                                                  <br>
                                                  [2021-03-16
                                                  19:05:05.227677] I
                                                  [MSGID: 132035]
                                                  [gf-history-changelog.c:837:gf_history_changelog]
                                                  0-gfchangelog:
                                                  Requesting historical
                                                  changelogs
                                                  [{start=1614666552},
                                                  {end=1615921505}]
                                                  <br>
                                                  [2021-03-16
                                                  19:05:05.227733] I
                                                  [MSGID: 132019]
                                                  [gf-history-changelog.c:755:gf_changelog_extract_min_max]
                                                  0-gfchangelog:
                                                  changelogs min max
                                                  [{min=1597342860},
                                                  {max=1615921502},
                                                  {total_changelogs=1300114}]
                                                  <br>
                                                  [2021-03-16
                                                  19:05:05.408567] E
                                                  [MSGID: 132009]
                                                  [gf-history-changelog.c:941:gf_history_changelog]
                                                  0-gfchangelog: wrong
                                                  result [{for=end},
                                                  {start=1615921502},
                                                  {idx=1300113}]
                                                  <br>
                                                  <br>
                                                  <br>
                                                  [2021-03-16
                                                  19:05:05.228092] I
                                                  [resource(worker
                                                  /data/storage_c/storage):1292:service_loop]
                                                  GLUSTER: Register time
                                                  [{time=1615921505}]<br>
                                                  [2021-03-16
                                                  19:05:05.228626] D
                                                  [repce(worker
                                                  /data/storage_c/storage):195:push]
                                                  RepceClient: call
                                                  124117:140500837320448:1615921505.23
                                                  keep_alive(None,) ...<br>
                                                  [2021-03-16
                                                  19:05:05.230076] D
                                                  [repce(worker
                                                  /data/storage_c/storage):215:__call__]
                                                  RepceClient: call
                                                  124117:140500837320448:1615921505.23
                                                  keep_alive -&gt; 1<br>
                                                  [2021-03-16
                                                  19:05:05.230693] D
                                                  [master(worker
                                                  /data/storage_c/storage):540:crawlwrap]
                                                  _GMaster: primary
                                                  master with volume id
cf94a8f2-324b-40b3-bf72-c3766100ea99 ...<br>
                                                  [2021-03-16
                                                  19:05:05.237607] I
                                                  [gsyncdstatus(worker
                                                  /data/storage_c/storage):281:set_active]
                                                  GeorepStatus: Worker
                                                  Status Change
                                                  [{status=Active}]<br>
                                                  [2021-03-16
                                                  19:05:05.242046] I
                                                  [gsyncdstatus(worker
                                                  /data/storage_c/storage):253:set_worker_crawl_status]
                                                  GeorepStatus: Crawl
                                                  Status Change
                                                  [{status=History
                                                  Crawl}]<br>
                                                  [2021-03-16
                                                  19:05:05.242450] I
                                                  [master(worker
                                                  /data/storage_c/storage):1559:crawl]
                                                  _GMaster: starting
                                                  history crawl
                                                  [{turns=1},
                                                  {stime=(1614666552,
                                                  0)},
                                                  {entry_stime=(1614664108,
                                                  0)},
                                                  {etime=1615921505}]<br>
                                                  [2021-03-16
                                                  19:05:05.244151] E
                                                  [resource(worker
                                                  /data/storage_c/storage):1312:service_loop]
                                                  GLUSTER: Changelog
                                                  History Crawl failed
                                                  [{error=[Errno 0]
                                                  Success}]<br>
                                                  [2021-03-16
                                                  19:05:05.394129] E
                                                  [resource(worker
                                                  /data/storage_a/storage):1312:service_loop]
                                                  GLUSTER: Changelog
                                                  History Crawl failed
                                                  [{error=[Errno 0]
                                                  Success}]<br>
                                                  [2021-03-16
                                                  19:05:05.408759] E
                                                  [resource(worker
                                                  /data/storage_b/storage):1312:service_loop]
                                                  GLUSTER: Changelog
                                                  History Crawl failed
                                                  [{error=[Errno 0]
                                                  Success}]<br>
                                                  [2021-03-16
                                                  19:05:06.158694] I
                                                  [monitor(monitor):228:monitor]
                                                  Monitor: worker died
                                                  in startup phase
                                                  [{brick=/data/storage_a/storage}]<br>
                                                  [2021-03-16
                                                  19:05:06.163052] I
                                                  [gsyncdstatus(monitor):248:set_worker_status]
                                                  GeorepStatus: Worker
                                                  Status Change
                                                  [{status=Faulty}]<br>
                                                  [2021-03-16
                                                  19:05:06.204464] I
                                                  [monitor(monitor):228:monitor]
                                                  Monitor: worker died
                                                  in startup phase
                                                  [{brick=/data/storage_b/storage}]<br>
                                                  [2021-03-16
                                                  19:05:06.208961] I
                                                  [gsyncdstatus(monitor):248:set_worker_status]
                                                  GeorepStatus: Worker
                                                  Status Change
                                                  [{status=Faulty}]<br>
                                                  [2021-03-16
                                                  19:05:06.220495] I
                                                  [monitor(monitor):228:monitor]
                                                  Monitor: worker died
                                                  in startup phase
                                                  [{brick=/data/storage_c/storage}]<br>
                                                  [2021-03-16
                                                  19:05:06.223947] I
                                                  [gsyncdstatus(monitor):248:set_worker_status]
                                                  GeorepStatus: Worker
                                                  Status Change
                                                  [{status=Faulty}]<br>
                                                  <br>
                                                  I confirmed NTP is
                                                  working: <br>
                                                  <br>
                                                  <br>
                                                  pcic-backup02 |
                                                  CHANGED | rc=0
                                                  &gt;&gt;<br>
                                                       remote          
                                                  refid      st t when
                                                  poll reach   delay  
                                                  offset  jitter<br>
==============================================================================<br>
                                                  +s216-232-132-95
                                                  68.69.221.61     2 u  
                                                  29 1024  377  
                                                  24.141    2.457  
                                                  1.081<br>
                                                  *<a rel="nofollow
                                                    noopener noreferrer"
href="http://yyz-1.ip.0xt.ca" target="_blank" moz-do-not-send="true">yyz-1.ip.0xt.ca</a>
                                                  206.108.0.131    2 u 
                                                  257 1024  377  
                                                  57.119   -0.084  
                                                  5.625<br>
                                                  +ip102.ip-198-27
                                                  192.168.10.254   2 u 
                                                  189 1024  377  
                                                  64.227   -3.012  
                                                  8.867<br>
                                                  <br>
                                                  storage03 | CHANGED |
                                                  rc=0 &gt;&gt;<br>
                                                       remote          
                                                  refid      st t when
                                                  poll reach   delay  
                                                  offset  jitter<br>
==============================================================================<br>
                                                  *198.161.203.36 
                                                  128.233.150.93   2 u  
                                                  36 1024  377  
                                                  16.055   -0.381  
                                                  0.318<br>
                                                  +s206-75-147-25.
                                                  192.168.10.254   2 u 
                                                  528 1024  377  
                                                  23.648   -6.196  
                                                  4.803<br>
                                                  +time.cloudflare
                                                  10.69.8.80       3 u 
                                                  121 1024  377   
                                                  2.408    0.507   0.791<br>
                                                  <br>
                                                  storage02 | CHANGED |
                                                  rc=0 &gt;&gt;<br>
                                                       remote          
                                                  refid      st t when
                                                  poll reach   delay  
                                                  offset  jitter<br>
==============================================================================<br>
                                                  *198.161.203.36 
                                                  128.233.150.93   2 u 
                                                  918 1024  377  
                                                  15.952    0.226  
                                                  0.197<br>
                                                  +linuxgeneration
                                                  16.164.40.197    2 u  
                                                  88 1024  377  
                                                  62.692   -1.160  
                                                  2.007<br>
                                                  +<a rel="nofollow
                                                    noopener noreferrer"
href="http://dns3.switch.ca" target="_blank" moz-do-not-send="true">dns3.switch.ca</a> 
                                                  206.108.0.131    2 u 
                                                  857 1024  377  
                                                  27.315    0.778  
                                                  0.483<br>
                                                  <br>
                                                  storage01 | CHANGED |
                                                  rc=0 &gt;&gt;<br>
                                                       remote          
                                                  refid      st t when
                                                  poll reach   delay  
                                                  offset  jitter<br>
==============================================================================<br>
                                                  +198.161.203.36 
                                                  128.233.150.93   2 u 
                                                  121 1024  377  
                                                  16.069    1.016  
                                                  0.195<br>
                                                  +<a rel="nofollow
                                                    noopener noreferrer"
href="http://zero.gotroot.ca" target="_blank" moz-do-not-send="true">zero.gotroot.ca</a>
                                                  30.114.5.31      2 u 
                                                  543 1024  377   
                                                  5.106   -2.462   4.923<br>
                                                  *<a rel="nofollow
                                                    noopener noreferrer"
href="http://ntp3.torix.ca" target="_blank" moz-do-not-send="true">ntp3.torix.ca</a>  
                                                  .PTP0.           1 u 
                                                  300 1024  377  
                                                  54.010    2.421 
                                                  15.182<br>
                                                  <br>
                                                  pcic-backup01 |
                                                  CHANGED | rc=0
                                                  &gt;&gt;<br>
                                                       remote          
                                                  refid      st t when
                                                  poll reach   delay  
                                                  offset  jitter<br>
==============================================================================<br>
                                                  *<a rel="nofollow
                                                    noopener noreferrer"
href="http://dns3.switch.ca" target="_blank" moz-do-not-send="true">dns3.switch.ca</a> 
                                                  206.108.0.131    2 u 
                                                  983 1024  377  
                                                  26.990    0.523  
                                                  1.389<br>
                                                  +<a rel="nofollow
                                                    noopener noreferrer"
href="http://dns2.switch.ca" target="_blank" moz-do-not-send="true">dns2.switch.ca</a> 
                                                  206.108.0.131    2 u 
                                                  689 1024  377  
                                                  26.975   -0.257  
                                                  0.467<br>
                                                  +64.ip-54-39-23.
                                                  214.176.184.39   2 u 
                                                  909 1024  377  
                                                  64.262   -0.604  
                                                  6.129<br>
                                                  <br>
                                                  And everything is
                                                  working on the same
                                                  version of gluster: <br>
                                                  <br>
                                                  pcic-backup02 |
                                                  CHANGED | rc=0
                                                  &gt;&gt;<br>
                                                  glusterfs 8.3<br>
                                                  pcic-backup01 |
                                                  CHANGED | rc=0
                                                  &gt;&gt;<br>
                                                  glusterfs 8.3<br>
                                                  storage02 | CHANGED |
                                                  rc=0 &gt;&gt;<br>
                                                  glusterfs 8.3<br>
                                                  storage01 | CHANGED |
                                                  rc=0 &gt;&gt;<br>
                                                  glusterfs 8.3<br>
                                                  storage03 | CHANGED |
                                                  rc=0 &gt;&gt;<br>
                                                  glusterfs 8.3<br>
                                                  <br>
                                                  SSH works, and the
                                                  backup user/group is
                                                  configured with
                                                  mountbroker: <br>
                                                  <br>
                                                  [root@storage01 ~]#
                                                  ssh -i
                                                  /root/.ssh/id_rsa <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:geoaccount@10.0.231.81"
                                                    target="_blank"
                                                    moz-do-not-send="true">
geoaccount@10.0.231.81</a> uname -a<br>
                                                  Linux pcic-backup01
                                                  3.10.0-1160.15.2.el7.x86_64
                                                  #1 SMP Wed Feb 3
                                                  15:06:38 UTC 2021
                                                  x86_64 x86_64 x86_64
                                                  GNU/Linux<br>
                                                  [root@storage01 ~]#
                                                  ssh -i
                                                  /root/.ssh/id_rsa <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
                                                    href="mailto:geoaccount@10.0.231.82"
                                                    target="_blank"
                                                    moz-do-not-send="true">
geoaccount@10.0.231.82</a> uname -a<br>
                                                  Linux pcic-backup02
                                                  3.10.0-1160.15.2.el7.x86_64
                                                  #1 SMP Wed Feb 3
                                                  15:06:38 UTC 2021
                                                  x86_64 x86_64 x86_64
                                                  GNU/Linux<br>
                                                  <br>
                                                  <br>
                                                  [root@pcic-backup01
                                                  ~]# grep geo
                                                  /etc/passwd<br>
geoaccount:x:1000:1000::/home/geoaccount:/bin/bash<br>
                                                  [root@pcic-backup01
                                                  ~]# grep geo
                                                  /etc/group<br>
geogroup:x:1000:geoaccount<br>
geoaccount:x:1001:geoaccount<br>
                                                  <br>
                                                  [root@pcic-backup01
                                                  ~]#
                                                  gluster-mountbroker
                                                  status<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
                                                  |     NODE    | NODE
                                                  STATUS |         MOUNT
                                                  ROOT        |   
                                                  GROUP     |         
                                                  USERS           |<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
                                                  | 10.0.231.82
                                                  |          UP |
                                                  /var/mountbroker-root(OK)
                                                  | geogroup(OK) |
                                                  geoaccount(pcic-backup) 
                                                  |<br>
                                                  |  localhost 
                                                  |          UP |
                                                  /var/mountbroker-root(OK)
                                                  | geogroup(OK) |
                                                  geoaccount(pcic-backup) 
                                                  |<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
                                                  <br>
                                                  <br>
                                                  <br>
                                                  <br>
                                                  So, then if I'm going
                                                  to have to resync,
                                                  what is the best way
                                                  to do this? <br>
                                                  <br>
                                                  With delete or delete
                                                  reset-sync-time ?   <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-starting_geo-replication#Deleting_a_Geo-replication_Session"
                                                    target="_blank"
                                                    moz-do-not-send="true">
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-starting_geo-replication#Deleting_a_Geo-replication_Session</a><br>
                                                  <br>
                                                  <br>
                                                  <br>
                                                  Erasing the index? So
                                                  I don't have to
                                                  transfer the files
                                                  again that are already
                                                  on the backup?
                                                  <br>
                                                  <ul>
                                                    <li><a rel="nofollow
                                                        noopener
                                                        noreferrer"
                                                        shape="rect"
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/sect-troubleshooting_geo-replication#Synchronization_Is_Not_Complete"
                                                        target="_blank"
moz-do-not-send="true">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/sect-troubleshooting_geo-replication#Synchronization_Is_Not_Complete</a>
                                                    </li>
                                                    <li><a rel="nofollow
                                                        noopener
                                                        noreferrer"
                                                        shape="rect"
href="https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Geo%20Replication/#best-practices"
                                                        target="_blank"
moz-do-not-send="true">https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Administrator%20Guide/Geo%20Replication/#best-practices</a><br>
                                                    </li>
                                                  </ul>
                                                  <br>
                                                  <br>
                                                  <br>
                                                  Is it possible to use
                                                  the special-sync-mode 
                                                  option from here: <a
                                                    rel="nofollow
                                                    noopener noreferrer"
                                                    shape="rect"
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-disaster_recovery"
                                                    target="_blank"
                                                    moz-do-not-send="true">
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/sect-disaster_recovery</a><br>
                                                  <br>
                                                  <br>
                                                  <br>
                                                  Thoughts? <br>
                                                  <br>
                                                  Thanks,<br>
                                                   -Matthew<br>
                                                  --<br>
                                                  <br>
                                                  <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yqt93720">
                                                    <div>On 3/12/21 3:31
                                                      PM, Strahil
                                                      Nikolov wrote:<br>
                                                    </div>
                                                    <blockquote
                                                      type="cite">
                                                      <div>Notice: This
                                                        message was sent
                                                        from outside the
                                                        University of
                                                        Victoria email
                                                        system. Please
                                                        be cautious with
                                                        links and
                                                        sensitive
                                                        information.
                                                      </div>
                                                      <br>
                                                      <div>Usually, when
                                                        I'm stuck - I
                                                        just start over.
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591681347">For
                                                          example, check
                                                          the
                                                          prerequisites:</div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591695583">-
                                                          Is ssh
                                                          available (no
                                                          firewall
                                                          blocking)</div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591712278">-
                                                          Is time sync
                                                          enabled
                                                          (ntp/chrony)</div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591726239">-
                                                          Is DNS ok on
                                                          all hosts
                                                          (including PTR
                                                          records)</div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591754957">-
                                                          Is the gluster
                                                          version the
                                                          same on all
                                                          nodes (primary
                                                          &amp;
                                                          secondary)</div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591754957"><br>
                                                        </div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591782389">Then
                                                          start over as
                                                          if the geo rep
                                                          was never
                                                          existing. For
                                                          example , stop
                                                          it and start
                                                          over with the
                                                          secondary
                                                          nodes's checks
                                                          (mountbroker,
                                                          user, group) .</div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591782389"><br>
                                                        </div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591841464">Most
                                                          probably
                                                          somwthing will
                                                          come up and
                                                          you will fix
                                                          it.</div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591841464"><br>
                                                        </div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591864006">In
                                                          worst case
                                                          scenario, you
                                                          will need to
                                                          clean ip the
                                                          geo-rep and
                                                          start fresh.</div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591864006"><br>
                                                        </div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591889862"><br>
                                                        </div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591889925">Best
                                                          Regards,</div>
                                                        <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615591895692">Strahil
                                                          Nikolov<br>
                                                          <br>
                                                          <blockquote>
                                                          <div>
                                                          <div>On Fri,
                                                          Mar 12, 2021
                                                          at 20:01,
                                                          Matthew
                                                          Benstead</div>
                                                          <div><a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:matthewb@uvic.ca"
target="_blank" moz-do-not-send="true">&lt;matthewb@uvic.ca&gt;</a>
                                                          wrote:</div>
                                                          </div>
                                                          <div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
                                                          <div>Hi
                                                          Strahil, <br>
                                                          <br>
                                                          Yes, SELinux
                                                          was put into
                                                          permissive
                                                          mode on the
                                                          secondary
                                                          nodes as well:
                                                          <br>
                                                          <br>
[root@pcic-backup01 ~]# sestatus | egrep -i  "^SELinux status|mode"<br>
                                                          SELinux
                                                          status:                
                                                          enabled<br>
                                                          Current
                                                          mode:                  
                                                          permissive<br>
                                                          Mode from
                                                          config
                                                          file:         
                                                          enforcing<br>
                                                          <br>
[root@pcic-backup02 ~]# sestatus | egrep -i  "^SELinux status|mode"<br>
                                                          SELinux
                                                          status:                
                                                          enabled<br>
                                                          Current
                                                          mode:                  
                                                          permissive<br>
                                                          Mode from
                                                          config
                                                          file:         
                                                          enforcing<br>
                                                          <br>
                                                          The secondary
                                                          server logs
                                                          didn't show
                                                          anything
                                                          interesting: <br>
                                                          <br>
                                                          gsyncd.log: <br>
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.81820]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:28.101819]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:28.107012]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:28.124567]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:28.128145]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:29.425739]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3184}]<br>
                                                          [2021-03-11
                                                          19:15:29.427448]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:29.433340]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3083}]<br>
                                                          [2021-03-11
                                                          19:15:29.434452]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3321}]<br>
                                                          [2021-03-11
                                                          19:15:29.434314]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:29.435575]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:29.439769]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3576}]<br>
                                                          [2021-03-11
                                                          19:15:29.440998]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:29.454745]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3262}]<br>
                                                          [2021-03-11
                                                          19:15:29.456192]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:32.594865]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:32.607815]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:32.647663]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:32.656280]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:32.668299]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:44.260689]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:44.271457]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:44.271883]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:44.279670]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:44.284261]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1116:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1116:connect</a>] GLUSTER: Mounting
                                                          gluster volume
                                                          locally...<br>
                                                          [2021-03-11
                                                          19:15:45.614280]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3419}]<br>
                                                          [2021-03-11
                                                          19:15:45.615622]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:45.617986]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3461}]<br>
                                                          [2021-03-11
                                                          19:15:45.618180]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3380}]<br>
                                                          [2021-03-11
                                                          19:15:45.619539]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:45.618999]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:45.620843]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3361}]<br>
                                                          [2021-03-11
                                                          19:15:45.621347]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1139:connect"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1139:connect</a>] GLUSTER: Mounted
                                                          gluster volume
[{duration=1.3604}]<br>
                                                          [2021-03-11
                                                          19:15:45.622179]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:45.622541]
                                                          I
                                                          [resource(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):1166:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):1166:service_loop</a>] GLUSTER:
                                                          slave
                                                          listening<br>
                                                          [2021-03-11
                                                          19:15:47.626054]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.91/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.91/data/storage_a/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:48.778399]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_c/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:48.778491]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.92/data/storage_c/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.92/data/storage_c/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:48.796854]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_a/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_a/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          [2021-03-11
                                                          19:15:48.800697]
                                                          I [repce(slave
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.93/data/storage_b/storage):96:service_loop"
target="_blank" moz-do-not-send="true">
10.0.231.93/data/storage_b/storage):96:service_loop</a>] RepceServer:
                                                          terminating on
                                                          reaching EOF.<br>
                                                          <br>
                                                          The mnt
                                                          geo-rep files
                                                          were also
                                                          uninteresting:
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.250150]
                                                          I [MSGID:
                                                          100030]
                                                          [glusterfsd.c:2689:main]
0-/usr/sbin/glusterfs: Started running version
                                                          [{arg=/usr/sbin/glusterfs},
                                                          {version=8.3},
{cmdlinestr=/usr/sbin/glusterfs --user-map-root=g<br>
                                                          eoaccount
                                                          --aux-gfid-mount
                                                          --acl
                                                          --log-level=INFO
--log-file=/var/log/glusterfs/geo-replication-slaves/storage_10.0.231.81_pcic-backup/mnt-10.0.231.93-data-storage_b-storage.log
--volfile-server=localhost --volf<br>
ile-id=pcic-backup --client-pid=-1
                                                          /var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.253485]
                                                          I
                                                          [glusterfsd.c:2424:daemonize]
                                                          0-glusterfs:
                                                          Pid of current
                                                          running
                                                          process is
                                                          157484<br>
                                                          [2021-03-11
                                                          19:15:28.267911]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=0}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.267984]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=1}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.268371]
                                                          I
                                                          [glusterfsd-mgmt.c:2170:mgmt_getspec_cbk]
                                                          0-glusterfs:
                                                          Received list
                                                          of available
                                                          volfile
                                                          servers:
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          href="http://10.0.231.82:24007"
target="_blank" moz-do-not-send="true">
10.0.231.82:24007</a> <br>
                                                          [2021-03-11
                                                          19:15:28.271729]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=2}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.271762]
                                                          I [MSGID:
                                                          101190]
                                                          [event-epoll.c:670:event_dispatch_epoll_worker]
                                                          0-epoll:
                                                          Started thread
                                                          with index
                                                          [{index=3}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.272223]
                                                          I [MSGID:
                                                          114020]
                                                          [client.c:2315:notify]
0-pcic-backup-client-0: parent translators are ready, attempting connect
                                                          on transport
                                                          []
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.275883]
                                                          I [MSGID:
                                                          114020]
                                                          [client.c:2315:notify]
0-pcic-backup-client-1: parent translators are ready, attempting connect
                                                          on transport
                                                          []
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.276154]
                                                          I
                                                          [rpc-clnt.c:1975:rpc_clnt_reconfig]
0-pcic-backup-client-0: changing port to 49153 (from 0)<br>
                                                          [2021-03-11
                                                          19:15:28.276193]
                                                          I
                                                          [socket.c:849:__socket_shutdown]
0-pcic-backup-client-0: intentional socket shutdown(13)<br>
                                                          Final graph:<br>
                                                          ...<br>
+------------------------------------------------------------------------------+<br>
                                                          [2021-03-11
                                                          19:15:28.282144]
                                                          I
                                                          [socket.c:849:__socket_shutdown]
0-pcic-backup-client-1: intentional socket shutdown(15)<br>
                                                          [2021-03-11
                                                          19:15:28.286536]
                                                          I [MSGID:
                                                          114057]
                                                          [client-handshake.c:1128:select_server_supported_programs]
0-pcic-backup-client-0: Using Program [{Program-name=GlusterFS 4.x v1},
                                                          {Num=1298437},
                                                          {Version=400}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.287208]
                                                          I [MSGID:
                                                          114046]
                                                          [client-handshake.c:857:client_setvolume_cbk]
0-pcic-backup-client-0: Connected, attached to remote volume
                                                          [{conn-name=pcic-backup-client-0},
{remote_subvol=/data/brick}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.290162]
                                                          I [MSGID:
                                                          114057]
                                                          [client-handshake.c:1128:select_server_supported_programs]
0-pcic-backup-client-1: Using Program [{Program-name=GlusterFS 4.x v1},
                                                          {Num=1298437},
                                                          {Version=400}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.291122]
                                                          I [MSGID:
                                                          114046]
                                                          [client-handshake.c:857:client_setvolume_cbk]
0-pcic-backup-client-1: Connected, attached to remote volume
                                                          [{conn-name=pcic-backup-client-1},
{remote_subvol=/data/brick}]
                                                          <br>
                                                          [2021-03-11
                                                          19:15:28.292703]
                                                          I
                                                          [fuse-bridge.c:5300:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24
                                                          kernel 7.23<br>
                                                          [2021-03-11
                                                          19:15:28.292730]
                                                          I
                                                          [fuse-bridge.c:5926:fuse_graph_sync]
                                                          0-fuse:
                                                          switched to
                                                          graph 0<br>
                                                          [2021-03-11
                                                          19:15:32.809518]
                                                          I
                                                          [fuse-bridge.c:6242:fuse_thread_proc]
                                                          0-fuse:
                                                          initiating
                                                          unmount of
                                                          /var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI<br>
                                                          [2021-03-11
                                                          19:15:32.810216]
                                                          W
                                                          [glusterfsd.c:1439:cleanup_and_exit]
(--&gt;/lib64/libpthread.so.0(+0x7ea5) [0x7ff56b175ea5]
                                                          --&gt;/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5)
[0x55664e67db45] --&gt;/usr/sbin/glusterfs(cleanup_and_exit+0x6b)
                                                          [0x55664e67d9ab]
                                                          ) 0-: received
                                                          signum (15),
                                                          shutting down
                                                          <br>
                                                          [2021-03-11
                                                          19:15:32.810253]
                                                          I
                                                          [fuse-bridge.c:7074:fini]
                                                          0-fuse:
                                                          Unmounting
                                                          '/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.<br>
                                                          [2021-03-11
                                                          19:15:32.810268]
                                                          I
                                                          [fuse-bridge.c:7079:fini]
                                                          0-fuse:
                                                          Closing fuse
                                                          connection to
'/var/mountbroker-root/user1000/mtpt-geoaccount-GmVoUI'.<br>
                                                          <br>
                                                          <br>
                                                          I'm really at
                                                          a loss for
                                                          where to go
                                                          from here, it
                                                          seems like
                                                          everything is
                                                          set up
                                                          correctly, and
                                                          it has been
                                                          working well
                                                          through the
                                                          7.x minor
                                                          versions, but
                                                          the jump to 8
                                                          has broken
                                                          something...<br>
                                                          <br>
                                                          There
                                                          definitely are
                                                          lots of
                                                          changelogs on
                                                          the servers
                                                          that fit into
                                                          the
                                                          timeframe.....
                                                          I haven't made
                                                          any writes to
                                                          the source
                                                          volume.... do
                                                          you think
                                                          that's the
                                                          problem? That
                                                          it needs some
                                                          new changelog
                                                          info to sync?
                                                          <br>
                                                          I had been
                                                          holding off
                                                          making any
                                                          writes in case
                                                          I needed to go
                                                          back to
                                                          Gluster7.9 -
                                                          not sure if
                                                          that's really
                                                          a good option
                                                          or not.
                                                          <br>
                                                          <br>
[root@storage01 changelogs]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | head; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          total 16G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  13K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343197<br>
                                                          -rw-r--r--. 1
                                                          root root  51K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343212<br>
                                                          -rw-r--r--. 1
                                                          root root  86K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343227<br>
                                                          -rw-r--r--. 1
                                                          root root  99K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343242<br>
                                                          -rw-r--r--. 1
                                                          root root  69K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343257<br>
                                                          -rw-r--r--. 1
                                                          root root  69K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343272<br>
                                                          -rw-r--r--. 1
                                                          root root  72K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343287<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          total 3.3G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  13K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343197<br>
                                                          -rw-r--r--. 1
                                                          root root  53K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343212<br>
                                                          -rw-r--r--. 1
                                                          root root  89K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343227<br>
                                                          -rw-r--r--. 1
                                                          root root  89K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343242<br>
                                                          -rw-r--r--. 1
                                                          root root  69K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343257<br>
                                                          -rw-r--r--. 1
                                                          root root  71K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343272<br>
                                                          -rw-r--r--. 1
                                                          root root  86K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343287<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          total 9.6G<br>
                                                          drw-------. 3
                                                          root root   16
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  16K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343199<br>
                                                          -rw-r--r--. 1
                                                          root root  71K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343214<br>
                                                          -rw-r--r--. 1
                                                          root root 122K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343229<br>
                                                          -rw-r--r--. 1
                                                          root root  73K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343244<br>
                                                          -rw-r--r--. 1
                                                          root root 100K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343259<br>
                                                          -rw-r--r--. 1
                                                          root root  95K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343274<br>
                                                          -rw-r--r--. 1
                                                          root root  92K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343289<br>
                                                          <br>
[root@storage01 changelogs]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | tail; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:33
                                                          CHANGELOG.1614663193<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663731<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663760<br>
                                                          -rw-r--r--. 1
                                                          root root  511
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664043<br>
                                                          -rw-r--r--. 1
                                                          root root  536
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664101<br>
                                                          -rw-r--r--. 1
                                                          root root 2.8K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664116<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666061<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666554<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663731<br>
                                                          -rw-r--r--. 1
                                                          root root  480
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663745<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663760<br>
                                                          -rw-r--r--. 1
                                                          root root  524
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664043<br>
                                                          -rw-r--r--. 1
                                                          root root  495
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664100<br>
                                                          -rw-r--r--. 1
                                                          root root 1.6K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664114<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666060<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666553<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663738<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663753<br>
                                                          -rw-r--r--. 1
                                                          root root  395
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664051<br>
                                                          -rw-r--r--. 1
                                                          root root  316
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664094<br>
                                                          -rw-r--r--. 1
                                                          root root 1.2K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664109<br>
                                                          -rw-r--r--. 1
                                                          root root  174
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664123<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666061<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666553<br>
                                                          drw-------. 2
                                                          root root    6
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   30
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
[root@storage02 ~]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | head; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          total 9.6G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root 4.2K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343193<br>
                                                          -rw-r--r--. 1
                                                          root root  32K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343208<br>
                                                          -rw-r--r--. 1
                                                          root root 107K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343223<br>
                                                          -rw-r--r--. 1
                                                          root root 120K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343238<br>
                                                          -rw-r--r--. 1
                                                          root root  72K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343253<br>
                                                          -rw-r--r--. 1
                                                          root root 111K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343268<br>
                                                          -rw-r--r--. 1
                                                          root root  91K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343283<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          total 16G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root 3.9K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343193<br>
                                                          -rw-r--r--. 1
                                                          root root  35K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343208<br>
                                                          -rw-r--r--. 1
                                                          root root  85K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343223<br>
                                                          -rw-r--r--. 1
                                                          root root 103K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343238<br>
                                                          -rw-r--r--. 1
                                                          root root  70K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343253<br>
                                                          -rw-r--r--. 1
                                                          root root  72K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343268<br>
                                                          -rw-r--r--. 1
                                                          root root  73K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343283<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          total 3.3G<br>
                                                          drw-------. 3
                                                          root root   16
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:51
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  21K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343202<br>
                                                          -rw-r--r--. 1
                                                          root root  75K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343217<br>
                                                          -rw-r--r--. 1
                                                          root root  92K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343232<br>
                                                          -rw-r--r--. 1
                                                          root root  77K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343247<br>
                                                          -rw-r--r--. 1
                                                          root root  66K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343262<br>
                                                          -rw-r--r--. 1
                                                          root root  84K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343277<br>
                                                          -rw-r--r--. 1
                                                          root root  81K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343292<br>
                                                          <br>
[root@storage02 ~]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | tail; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663734<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663749<br>
                                                          -rw-r--r--. 1
                                                          root root  395
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664052<br>
                                                          -rw-r--r--. 1
                                                          root root  316
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664096<br>
                                                          -rw-r--r--. 1
                                                          root root 1.2K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664111<br>
                                                          -rw-r--r--. 1
                                                          root root  174
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664126<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666056<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666560<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663735<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663749<br>
                                                          -rw-r--r--. 1
                                                          root root  511
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664052<br>
                                                          -rw-r--r--. 1
                                                          root root  316
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664096<br>
                                                          -rw-r--r--. 1
                                                          root root 1.8K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664111<br>
                                                          -rw-r--r--. 1
                                                          root root 1.4K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664126<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666060<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666556<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663738<br>
                                                          -rw-r--r--. 1
                                                          root root  521
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663752<br>
                                                          -rw-r--r--. 1
                                                          root root  524
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664042<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664057<br>
                                                          -rw-r--r--. 1
                                                          root root  536
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664102<br>
                                                          -rw-r--r--. 1
                                                          root root 1.6K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664117<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666057<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666550<br>
                                                          drw-------. 2
                                                          root root    6
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   30
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
                                                          <br>
[root@storage03 ~]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | head; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          total 3.4G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:50
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  19K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343201<br>
                                                          -rw-r--r--. 1
                                                          root root  66K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343215<br>
                                                          -rw-r--r--. 1
                                                          root root  91K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343230<br>
                                                          -rw-r--r--. 1
                                                          root root  82K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343245<br>
                                                          -rw-r--r--. 1
                                                          root root  64K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343259<br>
                                                          -rw-r--r--. 1
                                                          root root  75K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343274<br>
                                                          -rw-r--r--. 1
                                                          root root  81K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343289<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          total 9.6G<br>
                                                          drw-------. 3
                                                          root root   24
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:51
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root  19K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343201<br>
                                                          -rw-r--r--. 1
                                                          root root  80K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343215<br>
                                                          -rw-r--r--. 1
                                                          root root 119K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343230<br>
                                                          -rw-r--r--. 1
                                                          root root  65K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343244<br>
                                                          -rw-r--r--. 1
                                                          root root 100K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343259<br>
                                                          -rw-r--r--. 1
                                                          root root  95K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343274<br>
                                                          -rw-r--r--. 1
                                                          root root  92K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343289<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          total 16G<br>
                                                          drw-------. 3
                                                          root root   16
                                                          Mar  9 11:34
                                                          2021<br>
                                                          -rw-r--r--. 1
                                                          root root   51
                                                          Mar 12 09:51
                                                          CHANGELOG<br>
                                                          -rw-r--r--. 1
                                                          root root 3.9K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343193<br>
                                                          -rw-r--r--. 1
                                                          root root  35K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343208<br>
                                                          -rw-r--r--. 1
                                                          root root  85K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343223<br>
                                                          -rw-r--r--. 1
                                                          root root 103K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343238<br>
                                                          -rw-r--r--. 1
                                                          root root  70K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343253<br>
                                                          -rw-r--r--. 1
                                                          root root  71K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343268<br>
                                                          -rw-r--r--. 1
                                                          root root  73K
                                                          Aug 13  2020
                                                          CHANGELOG.1597343283<br>
                                                          <br>
[root@storage03 ~]# for dirs in {a,b,c}; do echo
                                                          "/data/storage_$dirs/storage/.glusterfs/changelogs";
                                                          ls -lh
                                                          /data/storage_$dirs/storage/.glusterfs/changelogs
                                                          | tail; echo
                                                          ""; done<br>
/data/storage_a/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:33
                                                          CHANGELOG.1614663183<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663740<br>
                                                          -rw-r--r--. 1
                                                          root root  521
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663755<br>
                                                          -rw-r--r--. 1
                                                          root root  524
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664049<br>
                                                          -rw-r--r--. 1
                                                          root root 1.9K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664106<br>
                                                          -rw-r--r--. 1
                                                          root root  174
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664121<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666051<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666559<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_b/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root  474
                                                          Mar  1 21:33
                                                          CHANGELOG.1614663182<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663739<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663753<br>
                                                          -rw-r--r--. 1
                                                          root root  395
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664049<br>
                                                          -rw-r--r--. 1
                                                          root root 1.4K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664106<br>
                                                          -rw-r--r--. 1
                                                          root root  174
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664120<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666063<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666557<br>
                                                          drw-------. 2
                                                          root root   10
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   38
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
/data/storage_c/storage/.glusterfs/changelogs<br>
                                                          -rw-r--r--. 1
                                                          root root  468
                                                          Mar  1 21:33
                                                          CHANGELOG.1614663183<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663740<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 21:42
                                                          CHANGELOG.1614663754<br>
                                                          -rw-r--r--. 1
                                                          root root  511
                                                          Mar  1 21:47
                                                          CHANGELOG.1614664048<br>
                                                          -rw-r--r--. 1
                                                          root root 2.0K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664105<br>
                                                          -rw-r--r--. 1
                                                          root root 1.4K
                                                          Mar  1 21:48
                                                          CHANGELOG.1614664120<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:20
                                                          CHANGELOG.1614666063<br>
                                                          -rw-r--r--. 1
                                                          root root   92
                                                          Mar  1 22:29
                                                          CHANGELOG.1614666556<br>
                                                          drw-------. 2
                                                          root root    6
                                                          May  7  2020
                                                          csnap<br>
                                                          drw-------. 2
                                                          root root   30
                                                          Aug 13  2020
                                                          htime<br>
                                                          <br>
                                                          Thanks,<br>
                                                           -Matthew<br>
                                                          <div>
                                                          <p>--<br>
                                                          Matthew
                                                          Benstead<br>
                                                          System
                                                          Administrator<br>
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="https://pacificclimate.org/"
target="_blank" moz-do-not-send="true">Pacific Climate Impacts
                                                          Consortium</a><br>
                                                          University of
                                                          Victoria, UH1<br>
                                                          PO Box 1800,
                                                          STN CSC<br>
                                                          Victoria, BC,
                                                          V8W 2Y2<br>
                                                          Phone:
                                                          +1-250-721-8432<br>
                                                          Email: <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:matthewb@uvic.ca"
target="_blank" moz-do-not-send="true">
matthewb@uvic.ca</a></p>
                                                          </div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yqt16014">
                                                          <div>On
                                                          3/11/21 11:37
                                                          PM, Strahil
                                                          Nikolov wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div>Notice:
                                                          This message
                                                          was sent from
                                                          outside the
                                                          University of
                                                          Victoria email
                                                          system. Please
                                                          be cautious
                                                          with links and
                                                          sensitive
                                                          information.
                                                          </div>
                                                          <br>
                                                          <div>Have you
                                                          checked the
                                                          secondary
                                                          volume nodes'
                                                          logs &amp;
                                                          SELINUX status
                                                          ?
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534613030"><br>
                                                          </div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534613170">Best
                                                          Regards,</div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534618885">Strahil
                                                          Nikolov<br>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633yMail_cursorElementTracker_1615534604587"><br>
                                                          <blockquote>
                                                          <div>
                                                          <div>On Thu,
                                                          Mar 11, 2021
                                                          at 21:36,
                                                          Matthew
                                                          Benstead</div>
                                                          <div><a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:matthewb@uvic.ca"
target="_blank" moz-do-not-send="true">&lt;matthewb@uvic.ca&gt;</a>
                                                          wrote:</div>
                                                          </div>
                                                          <div>
                                                          <div
id="gmail-m_634331480037335435yiv1091814412gmail-m_2443238309628334087yiv2558000633">
                                                          <div>Hi
                                                          Strahil, <br>
                                                          <br>
                                                          It looks like
                                                          perhaps the
                                                          changelog_log_level
                                                          and log_level
                                                          options? I've
                                                          set them to
                                                          debug:
                                                          <br>
                                                          <br>
[root@storage01 ~]# gluster volume geo-replication storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> config | egrep -i "log_level"<br>
changelog_log_level:INFO<br>
cli_log_level:INFO<br>
gluster_log_level:INFO<br>
                                                          log_level:INFO<br>
slave_gluster_log_level:INFO<br>
slave_log_level:INFO<br>
                                                          <br>
[root@storage01 ~]# gluster volume geo-replication storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> config changelog_log_level DEBUG<br>
geo-replication config updated successfully<br>
                                                          <br>
[root@storage01 ~]# gluster volume geo-replication storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> config log_level DEBUG<br>
geo-replication config updated successfully<br>
                                                          <br>
                                                          <br>
                                                          Then I
                                                          restarted
                                                          geo-replication:
                                                          <br>
                                                          <br>
[root@storage01 ~]# gluster volume geo-replication storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> start<br>
                                                          Starting
                                                          geo-replication
                                                          session
                                                          between
                                                          storage &amp;
                                                          <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
geoaccount@10.0.231.81::pcic-backup</a> has been successful<br>
[root@storage01 ~]# gluster volume geo-replication status <br>
                                                           <br>
                                                          MASTER NODE   
                                                          MASTER VOL   
                                                          MASTER
                                                          BRICK              
                                                          SLAVE USER   
SLAVE                                        SLAVE NODE   
                                                          STATUS            
                                                          CRAWL
                                                          STATUS   
                                                          LAST_SYNCED         
                                                          <br>
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A          
                                                          Initializing...   
N/A             N/A                 
                                                          <br>
[root@storage01 ~]# gluster volume geo-replication status <br>
                                                           <br>
                                                          MASTER NODE   
                                                          MASTER VOL   
                                                          MASTER
                                                          BRICK              
                                                          SLAVE USER   
SLAVE                                        SLAVE NODE    STATUS   
                                                          CRAWL
                                                          STATUS   
                                                          LAST_SYNCED         
                                                          <br>
---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.91   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.92   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_c/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_b/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A                 
                                                          <br>
                                                          10.0.231.93   
                                                          storage      
/data/storage_a/storage    geoaccount    <a rel="nofollow noopener
                                                          noreferrer"
                                                          shape="rect"
                                                          href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
target="_blank" moz-do-not-send="true">
ssh://geoaccount@10.0.231.81::pcic-backup</a>    N/A           Faulty   
N/A             N/A 
                                                          <br>
                                                          <br>
[root@storage01 ~]# gluster volume geo-replication storage <a
                                                          rel="nofollow
                                                          noopener
                                                          noreferrer"
                                                          shape="rect"