[Gluster-users] Geo-Replication not detecting changes

Thu Mar 30 12:53:01 UTC 2017

On Thu, Mar 30, 2017 at 04:58:53AM -0700, Jeremiah Rothschild wrote:
> Well, at anyrate, here you can see that both servers can talk on 49152/tcp:

Here is the list of ports that were explicitly opened for gfs:

24007/tcp
24008/tcp
24009/tcp
24010/tcp
49152/tcp
49153/tcp
111/tcp
111/udp

> [root at ill ~]# telnet aws 49152
> Trying 54.165.144.9...
> Connected to aws.
> Escape character is '^]'.
> 
> and
> 
> [root at aws jeremiah]# telnet ill 49152
> Trying 67.207.112.66...
> Connected to ill.
> Escape character is '^]'.
> 
> > Thanks and Regards,
> > Kotresh H R
> > 
> > ----- Original Message -----
> > > From: "Jeremiah Rothschild" <jeremiah at franz.com>
> > > To: "Kotresh Hiremath Ravishankar" <khiremat at redhat.com>
> > > Cc: gluster-users at gluster.org
> > > Sent: Thursday, March 30, 2017 1:16:03 PM
> > > Subject: Re: [Gluster-users] Geo-Replication not detecting changes
> > > 
> > > On Thu, Mar 30, 2017 at 12:51:23AM -0400, Kotresh Hiremath Ravishankar wrote:
> > > > Hi Jeremiah,
> > > 
> > > Hi Kotresh! Thanks for the follow-up!
> > > 
> > > > That's really strange. Please enable DEBUG logs for geo-replication as
> > > > below and send
> > > > us the logs under "/var/log/glusterfs/geo-replication/<mastervol>/*.log"
> > > > from master node
> > > > 
> > > > gluster vol geo-rep <mastervol> <slavehost>::<slavevol> config log-level
> > > > DEBUG
> > > 
> > > Ok.
> > > 
> > > I started from scratch & enabled debug level logging. The logs have been
> > > attached to Bugzilla #1327244.
> > > 
> > > > Geo-rep has two ways to detect changes.
> > > > 
> > > > 1. changelog (Changelog Crawl)
> > > > 2. xsync (Hybrid Crawl):
> > > >            This is good for initial sync. It has the limitation of not
> > > >            detecting unlinks and renames.
> > > >    So the slave would end up having unlinked files and renamed src file if
> > > >    it is used after initial sync.
> > > 
> > > FYI I did try changing the changelog_detector to xsync but it made no
> > > difference. Note that I also detailed this in the "Additional Info" section
> > > of the Bugzilla bug.
> > > 
> > > > Thanks and Regards,
> > > 
> > > Thanks again!
> > > 
> > > j
> > > 
> > > > Kotresh H R
> > > > 
> > > > ----- Original Message -----
> > > > > From: "Jeremiah Rothschild" <jeremiah at franz.com>
> > > > > To: gluster-users at gluster.org
> > > > > Sent: Wednesday, March 29, 2017 12:39:11 AM
> > > > > Subject: Re: [Gluster-users] Geo-Replication not detecting changes
> > > > > 
> > > > > Following up on my own thread...
> > > > > 
> > > > > I have spent hours and hours setting up, re-setting up, screwing with
> > > > > undocumented variables, upgrading from LTS to non-LTS, etc etc.
> > > > > 
> > > > > Nothing seems to give.
> > > > > 
> > > > > This is very much an out-of-the-box setup and core functionality just
> > > > > isn't
> > > > > working.
> > > > > 
> > > > > Can anyone throw me a bone here? Please? Do I file a bug for such an
> > > > > open-ended issue? Is everyone assuming I've just screwed a step up? I
> > > > > must
> > > > > say the documentation is pretty clear & simple. Do you want more logs?
> > > > > 
> > > > > If this is going to be a dead end then so be it but I at least need to
> > > > > make
> > > > > sure I've tried my hardest to get a working deployment.
> > > > > 
> > > > > Thanks for your time and understanding!
> > > > > 
> > > > > j
> > > > > 
> > > > > On Thu, Mar 23, 2017 at 11:47:03AM -0700, Jeremiah Rothschild wrote:
> > > > > > Hey all,
> > > > > > 
> > > > > > I have a vanilla geo-replication setup running. It is comprised of two
> > > > > > servers, both CentOS 7 and GlusterFS 3.8.10:
> > > > > > 
> > > > > > * server1: Local server. Master volume named "foo".
> > > > > > * server2: Remote server. Slave volume named "foo".
> > > > > > 
> > > > > > Everything went fine including the initial sync. However, no new
> > > > > > changes
> > > > > > are
> > > > > > being seen or synced.
> > > > > > 
> > > > > > Geo-rep status looks clean:
> > > > > > 
> > > > > > # gluster volume geo-replication foo server2.franz.com::foo status
> > > > > > MASTER NODE: server1.x.com
> > > > > > MASTER VOL: foo
> > > > > > MASTER BRICK: /gv0/foo
> > > > > > SLAVE USER: root
> > > > > > SLAVE NODE: server2.x.com::foo
> > > > > > STATUS: Active
> > > > > > CRAWL STATUS: Changelog Crawl
> > > > > > LAST_SYNCED: 2017-03-23 10:12:57
> > > > > > 
> > > > > > In the geo-rep master log, I see these being triggered:
> > > > > > 
> > > > > > # tail -n3
> > > > > > foo/ssh%3A%2F%2Froot%401.2.3.4%3Agluster%3A%2F%2F127.0.0.1%3Afoo.log
> > > > > > [2017-03-23 18:33:34.697525] I [master(/gv0/foo):534:crawlwrap]
> > > > > > _GMaster:
> > > > > > 20
> > > > > > crawls, 0 turns
> > > > > > [2017-03-23 18:34:37.441982] I [master(/gv0/foo):534:crawlwrap]
> > > > > > _GMaster:
> > > > > > 20
> > > > > > crawls, 0 turns
> > > > > > [2017-03-23 18:35:40.242851] I [master(/gv0/foo):534:crawlwrap]
> > > > > > _GMaster:
> > > > > > 20
> > > > > > crawls, 0 turns
> > > > > > 
> > > > > > I don't see any errors in any of the other logs.
> > > > > > 
> > > > > > Not sure what else to poke at here. What are the possible values for
> > > > > > the
> > > > > > "change_detector" config variable? Would it be worthwhile to test with
> > > > > > a
> > > > > > method other than "changelog"? Other thoughts/ideas?
> > > > > > 
> > > > > > Thanks in advance!
> > > > > > 
> > > > > > j
> > > > > > _______________________________________________
> > > > > > Gluster-users mailing list
> > > > > > Gluster-users at gluster.org
> > > > > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > > > > _______________________________________________
> > > > > Gluster-users mailing list
> > > > > Gluster-users at gluster.org
> > > > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > > > > 
> > > 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users