<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    Thanks Strahil, <br>
    <br>
    Right - I had come across your message in early January that v8 from
    the CentOS Sig was missing the SELinux rules, and had put SELinux
    into permissive mode after the upgrade when I saw denied messages in
    the audit logs. <br>
    <br>
    <font face="monospace">[root@storage01 ~]# sestatus | egrep
      "^SELinux status|[mM]ode"<br>
      SELinux status:                 enabled<br>
      Current mode:                   permissive<br>
      Mode from config file:          enforcing</font><br>
    <br>
    Yes - I am using an unprivileged user for georep:  <br>
    <br>
    <font face="monospace">[root@pcic-backup01 ~]# gluster-mountbroker
      status<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
      |     NODE    | NODE STATUS |         MOUNT ROOT        |   
      GROUP     |          USERS           |<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
      | 10.0.231.82 |          UP | /var/mountbroker-root(OK) |
      geogroup(OK) | geoaccount(pcic-backup)  |<br>
      |  localhost  |          UP | /var/mountbroker-root(OK) |
      geogroup(OK) | geoaccount(pcic-backup)  |<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
      <br>
      [root@pcic-backup02 ~]# gluster-mountbroker status<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
      |     NODE    | NODE STATUS |         MOUNT ROOT        |   
      GROUP     |          USERS           |<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
      | 10.0.231.81 |          UP | /var/mountbroker-root(OK) |
      geogroup(OK) | geoaccount(pcic-backup)  |<br>
      |  localhost  |          UP | /var/mountbroker-root(OK) |
      geogroup(OK) | geoaccount(pcic-backup)  |<br>
+-------------+-------------+---------------------------+--------------+--------------------------+<br>
      <br>
    </font>Thanks,<br>
     -Matthew<br>
    <div class="moz-signature"><font size="-1">
        <p>--<br>
          Matthew Benstead<br>
          <font size="-2">System Administrator<br>
            <a href="https://pacificclimate.org/">Pacific Climate
              Impacts Consortium</a><br>
            University of Victoria, UH1<br>
            PO Box 1800, STN CSC<br>
            Victoria, BC, V8W 2Y2<br>
            Phone: +1-250-721-8432<br>
            Email: <a class="moz-txt-link-abbreviated" href="mailto:matthewb@uvic.ca">matthewb@uvic.ca</a></font></p>
      </font>
    </div>
    <div class="moz-cite-prefix">On 3/10/21 2:11 PM, Strahil Nikolov
      wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:899679196.1091191.1615414262816@mail.yahoo.com">
      <div>
        Notice: This message was sent from outside the University of
        Victoria email system. Please be cautious with links and
        sensitive information.
      </div>
      <br>
      <div>I have tested georep on v8.3 and it was running quite well
        untill you involve SELINUX.
        <div id="yMail_cursorElementTracker_1615413996827"><br>
        </div>
        <div id="yMail_cursorElementTracker_1615413997227">Are you using
          SELINUX ?</div>
        <div id="yMail_cursorElementTracker_1615414006550">Are you using
          unprivileged user for the georep ?</div>
        <div id="yMail_cursorElementTracker_1615414240901"><br>
        </div>
        <div id="yMail_cursorElementTracker_1615414241153">Also, you can
check <a class="moz-txt-link-freetext" href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-troubleshooting_geo-replication">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-troubleshooting_geo-replication</a>
          .</div>
        <div id="yMail_cursorElementTracker_1615414031141"><br>
        </div>
        <div id="yMail_cursorElementTracker_1615414031310">Best Regards,</div>
        <div id="yMail_cursorElementTracker_1615414036428">Strahil
          Nikolov<br>
          <br>
          <blockquote>
            <div>
              <div>On Thu, Mar 11, 2021 at 0:03, Matthew Benstead</div>
              <div><a class="moz-txt-link-rfc2396E" href="mailto:matthewb@uvic.ca">&lt;matthewb@uvic.ca&gt;</a> wrote:</div>
            </div>
            <div>
              <div id="yiv8194796028">
                <div>Hello, <br>
                  <br>
                  I recently upgraded my Distributed-Replicate cluster
                  from Gluster 7.9 to 8.3 on CentOS7 using the CentOS
                  Storage SIG packages. I had geo-replication syncing
                  properly before the upgrade, but not it is not working
                  after.
                  <br>
                  <br>
                  After I had upgraded both master and slave clusters I
                  attempted to start geo-replication again, but it goes
                  to faulty quickly:
                  <br>
                  <br>
                  [root@storage01 ~]# gluster volume geo-replication
                  storage
                  <a rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    geoaccount@10.0.231.81::pcic-backup</a> start<br>
                  Starting geo-replication session between storage &amp;
                  <a rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    geoaccount@10.0.231.81::pcic-backup</a> has been
                  successful\<br>
                        <br>
                  [root@storage01 ~]# gluster volume geo-replication
                  status <br>
                   <br>
                  MASTER NODE    MASTER VOL    MASTER
                  BRICK               SLAVE USER   
                  SLAVE                                        SLAVE
                  NODE    STATUS    CRAWL STATUS    LAST_SYNCED         
                  <br>
---------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
                  10.0.231.91    storage      
                  /data/storage_a/storage    geoaccount    <a
                    rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    ssh://geoaccount@10.0.231.81::pcic-backup</a>   
                  N/A           Faulty    N/A            
                  N/A                 
                  <br>
                  10.0.231.91    storage      
                  /data/storage_c/storage    geoaccount    <a
                    rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    ssh://geoaccount@10.0.231.81::pcic-backup</a>   
                  N/A           Faulty    N/A            
                  N/A                 
                  <br>
                  10.0.231.91    storage      
                  /data/storage_b/storage    geoaccount    <a
                    rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    ssh://geoaccount@10.0.231.81::pcic-backup</a>   
                  N/A           Faulty    N/A            
                  N/A                 
                  <br>
                  10.0.231.92    storage      
                  /data/storage_b/storage    geoaccount    <a
                    rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    ssh://geoaccount@10.0.231.81::pcic-backup</a>   
                  N/A           Faulty    N/A            
                  N/A                 
                  <br>
                  10.0.231.92    storage      
                  /data/storage_a/storage    geoaccount    <a
                    rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    ssh://geoaccount@10.0.231.81::pcic-backup</a>   
                  N/A           Faulty    N/A            
                  N/A                 
                  <br>
                  10.0.231.92    storage      
                  /data/storage_c/storage    geoaccount    <a
                    rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    ssh://geoaccount@10.0.231.81::pcic-backup</a>   
                  N/A           Faulty    N/A            
                  N/A                 
                  <br>
                  10.0.231.93    storage      
                  /data/storage_c/storage    geoaccount    <a
                    rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    ssh://geoaccount@10.0.231.81::pcic-backup</a>   
                  N/A           Faulty    N/A            
                  N/A                 
                  <br>
                  10.0.231.93    storage      
                  /data/storage_b/storage    geoaccount    <a
                    rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    ssh://geoaccount@10.0.231.81::pcic-backup</a>   
                  N/A           Faulty    N/A            
                  N/A                 
                  <br>
                  10.0.231.93    storage      
                  /data/storage_a/storage    geoaccount    <a
                    rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:ssh://geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    ssh://geoaccount@10.0.231.81::pcic-backup</a>   
                  N/A           Faulty    N/A            
                  N/A                 
                  <br>
                  <br>
                  [root@storage01 ~]# gluster volume geo-replication
                  storage <a rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    geoaccount@10.0.231.81::pcic-backup</a> stop<br>
                  Stopping geo-replication session between storage &amp;
                  <a rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-abbreviated"
                    target="_blank"
                    href="mailto:geoaccount@10.0.231.81::pcic-backup"
                    moz-do-not-send="true">
                    geoaccount@10.0.231.81::pcic-backup</a> has been
                  successful<br>
                  <br>
                  <br>
                  I went through the gsyncd logs and see it attempts to
                  go back through the changlogs - which would make sense
                  - but fails:
                  <br>
                  <br>
                  [2021-03-10 19:18:42.165807] I
                  [gsyncdstatus(monitor):248:set_worker_status]
                  GeorepStatus: Worker Status Change
                  [{status=Initializing...}]<br>
                  [2021-03-10 19:18:42.166136] I
                  [monitor(monitor):160:monitor] Monitor: starting
                  gsyncd worker [{brick=/data/storage_a/storage},
                  {slave_node=10.0.231.81}]<br>
                  [2021-03-10 19:18:42.167829] I
                  [monitor(monitor):160:monitor] Monitor: starting
                  gsyncd worker [{brick=/data/storage_c/storage},
                  {slave_node=10.0.231.82}]<br>
                  [2021-03-10 19:18:42.172343] I
                  [gsyncdstatus(monitor):248:set_worker_status]
                  GeorepStatus: Worker Status Change
                  [{status=Initializing...}]<br>
                  [2021-03-10 19:18:42.172580] I
                  [monitor(monitor):160:monitor] Monitor: starting
                  gsyncd worker [{brick=/data/storage_b/storage},
                  {slave_node=10.0.231.82}]<br>
                  [2021-03-10 19:18:42.235574] I [resource(worker
                  /data/storage_c/storage):1387:connect_remote] SSH:
                  Initializing SSH connection between master and
                  slave...<br>
                  [2021-03-10 19:18:42.236613] I [resource(worker
                  /data/storage_a/storage):1387:connect_remote] SSH:
                  Initializing SSH connection between master and
                  slave...<br>
                  [2021-03-10 19:18:42.238614] I [resource(worker
                  /data/storage_b/storage):1387:connect_remote] SSH:
                  Initializing SSH connection between master and
                  slave...<br>
                  [2021-03-10 19:18:44.144856] I [resource(worker
                  /data/storage_b/storage):1436:connect_remote] SSH: SSH
                  connection between master and slave established.
                  [{duration=1.9059}]<br>
                  [2021-03-10 19:18:44.145065] I [resource(worker
                  /data/storage_b/storage):1116:connect] GLUSTER:
                  Mounting gluster volume locally...<br>
                  [2021-03-10 19:18:44.162873] I [resource(worker
                  /data/storage_a/storage):1436:connect_remote] SSH: SSH
                  connection between master and slave established.
                  [{duration=1.9259}]<br>
                  [2021-03-10 19:18:44.163412] I [resource(worker
                  /data/storage_a/storage):1116:connect] GLUSTER:
                  Mounting gluster volume locally...<br>
                  [2021-03-10 19:18:44.167506] I [resource(worker
                  /data/storage_c/storage):1436:connect_remote] SSH: SSH
                  connection between master and slave established.
                  [{duration=1.9316}]<br>
                  [2021-03-10 19:18:44.167746] I [resource(worker
                  /data/storage_c/storage):1116:connect] GLUSTER:
                  Mounting gluster volume locally...<br>
                  [2021-03-10 19:18:45.251372] I [resource(worker
                  /data/storage_b/storage):1139:connect] GLUSTER:
                  Mounted gluster volume [{duration=1.1062}]<br>
                  [2021-03-10 19:18:45.251583] I [subcmds(worker
                  /data/storage_b/storage):84:subcmd_worker]
                  &lt;top&gt;: Worker spawn successful. Acknowledging
                  back to monitor<br>
                  [2021-03-10 19:18:45.271950] I [resource(worker
                  /data/storage_c/storage):1139:connect] GLUSTER:
                  Mounted gluster volume [{duration=1.1041}]<br>
                  [2021-03-10 19:18:45.272118] I [subcmds(worker
                  /data/storage_c/storage):84:subcmd_worker]
                  &lt;top&gt;: Worker spawn successful. Acknowledging
                  back to monitor<br>
                  [2021-03-10 19:18:45.275180] I [resource(worker
                  /data/storage_a/storage):1139:connect] GLUSTER:
                  Mounted gluster volume [{duration=1.1116}]<br>
                  [2021-03-10 19:18:45.275361] I [subcmds(worker
                  /data/storage_a/storage):84:subcmd_worker]
                  &lt;top&gt;: Worker spawn successful. Acknowledging
                  back to monitor<br>
                  [2021-03-10 19:18:47.265618] I [master(worker
                  /data/storage_b/storage):1645:register] _GMaster:
                  Working dir
[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_b-storage}]<br>
                  [2021-03-10 19:18:47.265954] I [resource(worker
                  /data/storage_b/storage):1292:service_loop] GLUSTER:
                  Register time [{time=1615403927}]<br>
                  [2021-03-10 19:18:47.276746] I [gsyncdstatus(worker
                  /data/storage_b/storage):281:set_active] GeorepStatus:
                  Worker Status Change [{status=Active}]<br>
                  [2021-03-10 19:18:47.281194] I [gsyncdstatus(worker
                  /data/storage_b/storage):253:set_worker_crawl_status]
                  GeorepStatus: Crawl Status Change [{status=History
                  Crawl}]<br>
                  [2021-03-10 19:18:47.281404] I [master(worker
                  /data/storage_b/storage):1559:crawl] _GMaster:
                  starting history crawl [{turns=1}, {stime=(1614666552,
                  0)}, {entry_stime=(1614664113, 0)},
                  {etime=1615403927}]<br>
                  [2021-03-10 19:18:47.285340] I [master(worker
                  /data/storage_c/storage):1645:register] _GMaster:
                  Working dir
[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_c-storage}]<br>
                  [2021-03-10 19:18:47.285579] I [resource(worker
                  /data/storage_c/storage):1292:service_loop] GLUSTER:
                  Register time [{time=1615403927}]<br>
                  [2021-03-10 19:18:47.287383] I [master(worker
                  /data/storage_a/storage):1645:register] _GMaster:
                  Working dir
[{path=/var/lib/misc/gluster/gsyncd/storage_10.0.231.81_pcic-backup/data-storage_a-storage}]<br>
                  [2021-03-10 19:18:47.287697] I [resource(worker
                  /data/storage_a/storage):1292:service_loop] GLUSTER:
                  Register time [{time=1615403927}]<br>
                  [2021-03-10 19:18:47.298415] I [gsyncdstatus(worker
                  /data/storage_c/storage):281:set_active] GeorepStatus:
                  Worker Status Change [{status=Active}]<br>
                  [2021-03-10 19:18:47.301342] I [gsyncdstatus(worker
                  /data/storage_a/storage):281:set_active] GeorepStatus:
                  Worker Status Change [{status=Active}]<br>
                  [2021-03-10 19:18:47.304183] I [gsyncdstatus(worker
                  /data/storage_c/storage):253:set_worker_crawl_status]
                  GeorepStatus: Crawl Status Change [{status=History
                  Crawl}]<br>
                  [2021-03-10 19:18:47.304418] I [master(worker
                  /data/storage_c/storage):1559:crawl] _GMaster:
                  starting history crawl [{turns=1}, {stime=(1614666552,
                  0)}, {entry_stime=(1614664108, 0)},
                  {etime=1615403927}]<br>
                  [2021-03-10 19:18:47.305294] E [resource(worker
                  /data/storage_c/storage):1312:service_loop] GLUSTER:
                  Changelog History Crawl failed [{error=[Errno 0]
                  Success}]<br>
                  [2021-03-10 19:18:47.308124] I [gsyncdstatus(worker
                  /data/storage_a/storage):253:set_worker_crawl_status]
                  GeorepStatus: Crawl Status Change [{status=History
                  Crawl}]<br>
                  [2021-03-10 19:18:47.308509] I [master(worker
                  /data/storage_a/storage):1559:crawl] _GMaster:
                  starting history crawl [{turns=1}, {stime=(1614666553,
                  0)}, {entry_stime=(1614664115, 0)},
                  {etime=1615403927}]<br>
                  [2021-03-10 19:18:47.357470] E [resource(worker
                  /data/storage_b/storage):1312:service_loop] GLUSTER:
                  Changelog History Crawl failed [{error=[Errno 0]
                  Success}]<br>
                  [2021-03-10 19:18:47.383949] E [resource(worker
                  /data/storage_a/storage):1312:service_loop] GLUSTER:
                  Changelog History Crawl failed [{error=[Errno 0]
                  Success}]<br>
                  [2021-03-10 19:18:48.255340] I
                  [monitor(monitor):228:monitor] Monitor: worker died in
                  startup phase [{brick=/data/storage_b/storage}]<br>
                  [2021-03-10 19:18:48.260052] I
                  [gsyncdstatus(monitor):248:set_worker_status]
                  GeorepStatus: Worker Status Change [{status=Faulty}]<br>
                  [2021-03-10 19:18:48.275651] I
                  [monitor(monitor):228:monitor] Monitor: worker died in
                  startup phase [{brick=/data/storage_c/storage}]<br>
                  [2021-03-10 19:18:48.278064] I
                  [monitor(monitor):228:monitor] Monitor: worker died in
                  startup phase [{brick=/data/storage_a/storage}]<br>
                  [2021-03-10 19:18:48.280453] I
                  [gsyncdstatus(monitor):248:set_worker_status]
                  GeorepStatus: Worker Status Change [{status=Faulty}]<br>
                  [2021-03-10 19:18:48.282274] I
                  [gsyncdstatus(monitor):248:set_worker_status]
                  GeorepStatus: Worker Status Change [{status=Faulty}]<br>
                  [2021-03-10 19:18:58.275702] I
                  [gsyncdstatus(monitor):248:set_worker_status]
                  GeorepStatus: Worker Status Change
                  [{status=Initializing...}]<br>
                  [2021-03-10 19:18:58.276041] I
                  [monitor(monitor):160:monitor] Monitor: starting
                  gsyncd worker [{brick=/data/storage_b/storage},
                  {slave_node=10.0.231.82}]<br>
                  [2021-03-10 19:18:58.296252] I
                  [gsyncdstatus(monitor):248:set_worker_status]
                  GeorepStatus: Worker Status Change
                  [{status=Initializing...}]<br>
                  [2021-03-10 19:18:58.296506] I
                  [monitor(monitor):160:monitor] Monitor: starting
                  gsyncd worker [{brick=/data/storage_c/storage},
                  {slave_node=10.0.231.82}]<br>
                  [2021-03-10 19:18:58.301290] I
                  [gsyncdstatus(monitor):248:set_worker_status]
                  GeorepStatus: Worker Status Change
                  [{status=Initializing...}]<br>
                  [2021-03-10 19:18:58.301521] I
                  [monitor(monitor):160:monitor] Monitor: starting
                  gsyncd worker [{brick=/data/storage_a/storage},
                  {slave_node=10.0.231.81}]<br>
                  [2021-03-10 19:18:58.345817] I [resource(worker
                  /data/storage_b/storage):1387:connect_remote] SSH:
                  Initializing SSH connection between master and
                  slave...<br>
                  [2021-03-10 19:18:58.361268] I [resource(worker
                  /data/storage_c/storage):1387:connect_remote] SSH:
                  Initializing SSH connection between master and
                  slave...<br>
                  [2021-03-10 19:18:58.367985] I [resource(worker
                  /data/storage_a/storage):1387:connect_remote] SSH:
                  Initializing SSH connection between master and
                  slave...<br>
                  [2021-03-10 19:18:59.115143] I
                  [subcmds(monitor-status):29:subcmd_monitor_status]
                  &lt;top&gt;: Monitor Status Change [{status=Stopped}]<br>
                  <br>
                  It seems like there is an issue selecting the
                  changelogs - perhaps similar to this issue?
                  <a rel="nofollow noopener noreferrer"
                    class="yiv8194796028moz-txt-link-freetext"
                    target="_blank"
                    href="https://github.com/gluster/glusterfs/issues/1766"
                    moz-do-not-send="true">
                    https://github.com/gluster/glusterfs/issues/1766</a><br>
                  <br>
                  [root@storage01 storage_10.0.231.81_pcic-backup]# cat
                  changes-data-storage_a-storage.log<br>
                  [2021-03-10 19:18:45.284764] I [MSGID: 132028]
                  [gf-changelog.c:577:gf_changelog_register_generic]
                  0-gfchangelog: Registering brick
                  [{brick=/data/storage_a/storage}, {notify_filter=1}]
                  <br>
                  [2021-03-10 19:18:45.285275] I [MSGID: 101190]
                  [event-epoll.c:670:event_dispatch_epoll_worker]
                  0-epoll: Started thread with index [{index=3}]
                  <br>
                  [2021-03-10 19:18:45.285269] I [MSGID: 101190]
                  [event-epoll.c:670:event_dispatch_epoll_worker]
                  0-epoll: Started thread with index [{index=2}]
                  <br>
                  [2021-03-10 19:18:45.286615] I
                  [socket.c:929:__socket_server_bind]
                  0-socket.gfchangelog: closing (AF_UNIX) reuse check
                  socket 21<br>
                  [2021-03-10 19:18:47.308607] I [MSGID: 132035]
                  [gf-history-changelog.c:837:gf_history_changelog]
                  0-gfchangelog: Requesting historical changelogs
                  [{start=1614666553}, {end=1615403927}]
                  <br>
                  [2021-03-10 19:18:47.308659] I [MSGID: 132019]
                  [gf-history-changelog.c:755:gf_changelog_extract_min_max]
                  0-gfchangelog: changelogs min max [{min=1597342860},
                  {max=1615403927}, {total_changelogs=1250878}]
                  <br>
                  [2021-03-10 19:18:47.383774] E [MSGID: 132009]
                  [gf-history-changelog.c:941:gf_history_changelog]
                  0-gfchangelog: wrong result [{for=end},
                  {start=1615403927}, {idx=1250877}]
                  <br>
                  <br>
                  [root@storage01 storage_10.0.231.81_pcic-backup]# tail
                  -7 changes-data-storage_b-storage.log<br>
                  [2021-03-10 19:18:45.263211] I [MSGID: 101190]
                  [event-epoll.c:670:event_dispatch_epoll_worker]
                  0-epoll: Started thread with index [{index=3}]
                  <br>
                  [2021-03-10 19:18:45.263151] I [MSGID: 132028]
                  [gf-changelog.c:577:gf_changelog_register_generic]
                  0-gfchangelog: Registering brick
                  [{brick=/data/storage_b/storage}, {notify_filter=1}]
                  <br>
                  [2021-03-10 19:18:45.263294] I [MSGID: 101190]
                  [event-epoll.c:670:event_dispatch_epoll_worker]
                  0-epoll: Started thread with index [{index=2}]
                  <br>
                  [2021-03-10 19:18:45.264598] I
                  [socket.c:929:__socket_server_bind]
                  0-socket.gfchangelog: closing (AF_UNIX) reuse check
                  socket 23<br>
                  [2021-03-10 19:18:47.281499] I [MSGID: 132035]
                  [gf-history-changelog.c:837:gf_history_changelog]
                  0-gfchangelog: Requesting historical changelogs
                  [{start=1614666552}, {end=1615403927}]
                  <br>
                  [2021-03-10 19:18:47.281551] I [MSGID: 132019]
                  [gf-history-changelog.c:755:gf_changelog_extract_min_max]
                  0-gfchangelog: changelogs min max [{min=1597342860},
                  {max=1615403927}, {total_changelogs=1258258}]
                  <br>
                  [2021-03-10 19:18:47.357244] E [MSGID: 132009]
                  [gf-history-changelog.c:941:gf_history_changelog]
                  0-gfchangelog: wrong result [{for=end},
                  {start=1615403927}, {idx=1258257}]
                  <br>
                  <br>
                  Any ideas on where to debug this? I'd prefer not to
                  have to remove and re-sync everything as there is
                  about 240TB on the cluster...<br>
                  <br>
                  Thanks,<br>
                   -Matthew<br>
                </div>
              </div>
              ________<br>
              <br>
              <br>
              <br>
              Community Meeting Calendar:<br>
              <br>
              Schedule -<br>
              Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
              Bridge: <a href="https://meet.google.com/cpu-eiue-hvk"
                target="_blank" moz-do-not-send="true">https://meet.google.com/cpu-eiue-hvk</a><br>
              Gluster-users mailing list<br>
              <a href="mailto:Gluster-users@gluster.org"
                moz-do-not-send="true">Gluster-users@gluster.org</a><br>
              <a
                href="https://lists.gluster.org/mailman/listinfo/gluster-users"
                target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
            </div>
          </blockquote>
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>