[Gluster-users] Geo-Replication not detecting changes

Thu Mar 30 11:58:53 UTC 2017

On Thu, Mar 30, 2017 at 05:49:32AM -0400, Kotresh Hiremath Ravishankar wrote:
> Hi Jeremiah,
> 
> I believe the bug ID is #1437244 and not #1327244.

Oops! You are correct.

> >From the geo-rep logs, the master volume is failed with "Transport Endpoint Not Connected"
> ...
> [2017-03-30 07:40:57.150348] E [resource(/gv0/foo):234:errlog] Popen: command "/usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication/foo/ssh%3A%2F%2Froot%4054.165.144.9%3Agluster%3A%2F%2F127.0.0.1%3Afoo.%2Fgv0%2Ffoo.gluster.log --volfile-server=localhost --volfile-id=foo --client-pid=-1 /tmp/gsyncd-aux-mount-K1j3ZD" returned with 107
> ..
> 
> Could you try flushing iptables on both master and slave nodes and check again?
> #iptables -F

Done. I then restarted glusterd on both servers and waited for a sync to
happen but there is no change.

Also, I believe that networking was verified as OK, because the initial sync
worked? Is that not true?

Well, at anyrate, here you can see that both servers can talk on 49152/tcp:

[root at ill ~]# telnet aws 49152
Trying 54.165.144.9...
Connected to aws.
Escape character is '^]'.

and

[root at aws jeremiah]# telnet ill 49152
Trying 67.207.112.66...
Connected to ill.
Escape character is '^]'.

> Thanks and Regards,
> Kotresh H R
> 
> ----- Original Message -----
> > From: "Jeremiah Rothschild" <jeremiah at franz.com>
> > To: "Kotresh Hiremath Ravishankar" <khiremat at redhat.com>
> > Cc: gluster-users at gluster.org
> > Sent: Thursday, March 30, 2017 1:16:03 PM
> > Subject: Re: [Gluster-users] Geo-Replication not detecting changes
> > 
> > On Thu, Mar 30, 2017 at 12:51:23AM -0400, Kotresh Hiremath Ravishankar wrote:
> > > Hi Jeremiah,
> > 
> > Hi Kotresh! Thanks for the follow-up!
> > 
> > > That's really strange. Please enable DEBUG logs for geo-replication as
> > > below and send
> > > us the logs under "/var/log/glusterfs/geo-replication/<mastervol>/*.log"
> > > from master node
> > > 
> > > gluster vol geo-rep <mastervol> <slavehost>::<slavevol> config log-level
> > > DEBUG
> > 
> > Ok.
> > 
> > I started from scratch & enabled debug level logging. The logs have been
> > attached to Bugzilla #1327244.
> > 
> > > Geo-rep has two ways to detect changes.
> > > 
> > > 1. changelog (Changelog Crawl)
> > > 2. xsync (Hybrid Crawl):
> > >            This is good for initial sync. It has the limitation of not
> > >            detecting unlinks and renames.
> > >    So the slave would end up having unlinked files and renamed src file if
> > >    it is used after initial sync.
> > 
> > FYI I did try changing the changelog_detector to xsync but it made no
> > difference. Note that I also detailed this in the "Additional Info" section
> > of the Bugzilla bug.
> > 
> > > Thanks and Regards,
> > 
> > Thanks again!
> > 
> > j
> > 
> > > Kotresh H R
> > > 
> > > ----- Original Message -----
> > > > From: "Jeremiah Rothschild" <jeremiah at franz.com>
> > > > To: gluster-users at gluster.org
> > > > Sent: Wednesday, March 29, 2017 12:39:11 AM
> > > > Subject: Re: [Gluster-users] Geo-Replication not detecting changes
> > > > 
> > > > Following up on my own thread...
> > > > 
> > > > I have spent hours and hours setting up, re-setting up, screwing with
> > > > undocumented variables, upgrading from LTS to non-LTS, etc etc.
> > > > 
> > > > Nothing seems to give.
> > > > 
> > > > This is very much an out-of-the-box setup and core functionality just
> > > > isn't
> > > > working.
> > > > 
> > > > Can anyone throw me a bone here? Please? Do I file a bug for such an
> > > > open-ended issue? Is everyone assuming I've just screwed a step up? I
> > > > must
> > > > say the documentation is pretty clear & simple. Do you want more logs?
> > > > 
> > > > If this is going to be a dead end then so be it but I at least need to
> > > > make
> > > > sure I've tried my hardest to get a working deployment.
> > > > 
> > > > Thanks for your time and understanding!
> > > > 
> > > > j
> > > > 
> > > > On Thu, Mar 23, 2017 at 11:47:03AM -0700, Jeremiah Rothschild wrote:
> > > > > Hey all,
> > > > > 
> > > > > I have a vanilla geo-replication setup running. It is comprised of two
> > > > > servers, both CentOS 7 and GlusterFS 3.8.10:
> > > > > 
> > > > > * server1: Local server. Master volume named "foo".
> > > > > * server2: Remote server. Slave volume named "foo".
> > > > > 
> > > > > Everything went fine including the initial sync. However, no new
> > > > > changes
> > > > > are
> > > > > being seen or synced.
> > > > > 
> > > > > Geo-rep status looks clean:
> > > > > 
> > > > > # gluster volume geo-replication foo server2.franz.com::foo status
> > > > > MASTER NODE: server1.x.com
> > > > > MASTER VOL: foo
> > > > > MASTER BRICK: /gv0/foo
> > > > > SLAVE USER: root
> > > > > SLAVE NODE: server2.x.com::foo
> > > > > STATUS: Active
> > > > > CRAWL STATUS: Changelog Crawl
> > > > > LAST_SYNCED: 2017-03-23 10:12:57
> > > > > 
> > > > > In the geo-rep master log, I see these being triggered:
> > > > > 
> > > > > # tail -n3
> > > > > foo/ssh%3A%2F%2Froot%401.2.3.4%3Agluster%3A%2F%2F127.0.0.1%3Afoo.log
> > > > > [2017-03-23 18:33:34.697525] I [master(/gv0/foo):534:crawlwrap]
> > > > > _GMaster:
> > > > > 20
> > > > > crawls, 0 turns
> > > > > [2017-03-23 18:34:37.441982] I [master(/gv0/foo):534:crawlwrap]
> > > > > _GMaster:
> > > > > 20
> > > > > crawls, 0 turns
> > > > > [2017-03-23 18:35:40.242851] I [master(/gv0/foo):534:crawlwrap]
> > > > > _GMaster:
> > > > > 20
> > > > > crawls, 0 turns
> > > > > 
> > > > > I don't see any errors in any of the other logs.
> > > > > 
> > > > > Not sure what else to poke at here. What are the possible values for
> > > > > the
> > > > > "change_detector" config variable? Would it be worthwhile to test with
> > > > > a
> > > > > method other than "changelog"? Other thoughts/ideas?
> > > > > 
> > > > > Thanks in advance!
> > > > > 
> > > > > j
> > > > > _______________________________________________
> > > > > Gluster-users mailing list
> > > > > Gluster-users at gluster.org
> > > > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > > > 
> > 