[Gluster-users] Geo-Replication not detecting changes

Thu Mar 30 09:49:32 UTC 2017

Hi Jeremiah,

I believe the bug ID is #1437244 and not #1327244.
>From the geo-rep logs, the master volume is failed with "Transport Endpoint Not Connected"
...
[2017-03-30 07:40:57.150348] E [resource(/gv0/foo):234:errlog] Popen: command "/usr/sbin/glusterfs --aux-gfid-mount --acl --log-file=/var/log/glusterfs/geo-replication/foo/ssh%3A%2F%2Froot%4054.165.144.9%3Agluster%3A%2F%2F127.0.0.1%3Afoo.%2Fgv0%2Ffoo.gluster.log --volfile-server=localhost --volfile-id=foo --client-pid=-1 /tmp/gsyncd-aux-mount-K1j3ZD" returned with 107
..

Could you try flushing iptables on both master and slave nodes and check again?
#iptables -F

Thanks and Regards,
Kotresh H R

----- Original Message -----
> From: "Jeremiah Rothschild" <jeremiah at franz.com>
> To: "Kotresh Hiremath Ravishankar" <khiremat at redhat.com>
> Cc: gluster-users at gluster.org
> Sent: Thursday, March 30, 2017 1:16:03 PM
> Subject: Re: [Gluster-users] Geo-Replication not detecting changes
> 
> On Thu, Mar 30, 2017 at 12:51:23AM -0400, Kotresh Hiremath Ravishankar wrote:
> > Hi Jeremiah,
> 
> Hi Kotresh! Thanks for the follow-up!
> 
> > That's really strange. Please enable DEBUG logs for geo-replication as
> > below and send
> > us the logs under "/var/log/glusterfs/geo-replication/<mastervol>/*.log"
> > from master node
> > 
> > gluster vol geo-rep <mastervol> <slavehost>::<slavevol> config log-level
> > DEBUG
> 
> Ok.
> 
> I started from scratch & enabled debug level logging. The logs have been
> attached to Bugzilla #1327244.
> 
> > Geo-rep has two ways to detect changes.
> > 
> > 1. changelog (Changelog Crawl)
> > 2. xsync (Hybrid Crawl):
> >            This is good for initial sync. It has the limitation of not
> >            detecting unlinks and renames.
> >    So the slave would end up having unlinked files and renamed src file if
> >    it is used after initial sync.
> 
> FYI I did try changing the changelog_detector to xsync but it made no
> difference. Note that I also detailed this in the "Additional Info" section
> of the Bugzilla bug.
> 
> > Thanks and Regards,
> 
> Thanks again!
> 
> j
> 
> > Kotresh H R
> > 
> > ----- Original Message -----
> > > From: "Jeremiah Rothschild" <jeremiah at franz.com>
> > > To: gluster-users at gluster.org
> > > Sent: Wednesday, March 29, 2017 12:39:11 AM
> > > Subject: Re: [Gluster-users] Geo-Replication not detecting changes
> > > 
> > > Following up on my own thread...
> > > 
> > > I have spent hours and hours setting up, re-setting up, screwing with
> > > undocumented variables, upgrading from LTS to non-LTS, etc etc.
> > > 
> > > Nothing seems to give.
> > > 
> > > This is very much an out-of-the-box setup and core functionality just
> > > isn't
> > > working.
> > > 
> > > Can anyone throw me a bone here? Please? Do I file a bug for such an
> > > open-ended issue? Is everyone assuming I've just screwed a step up? I
> > > must
> > > say the documentation is pretty clear & simple. Do you want more logs?
> > > 
> > > If this is going to be a dead end then so be it but I at least need to
> > > make
> > > sure I've tried my hardest to get a working deployment.
> > > 
> > > Thanks for your time and understanding!
> > > 
> > > j
> > > 
> > > On Thu, Mar 23, 2017 at 11:47:03AM -0700, Jeremiah Rothschild wrote:
> > > > Hey all,
> > > > 
> > > > I have a vanilla geo-replication setup running. It is comprised of two
> > > > servers, both CentOS 7 and GlusterFS 3.8.10:
> > > > 
> > > > * server1: Local server. Master volume named "foo".
> > > > * server2: Remote server. Slave volume named "foo".
> > > > 
> > > > Everything went fine including the initial sync. However, no new
> > > > changes
> > > > are
> > > > being seen or synced.
> > > > 
> > > > Geo-rep status looks clean:
> > > > 
> > > > # gluster volume geo-replication foo server2.franz.com::foo status
> > > > MASTER NODE: server1.x.com
> > > > MASTER VOL: foo
> > > > MASTER BRICK: /gv0/foo
> > > > SLAVE USER: root
> > > > SLAVE NODE: server2.x.com::foo
> > > > STATUS: Active
> > > > CRAWL STATUS: Changelog Crawl
> > > > LAST_SYNCED: 2017-03-23 10:12:57
> > > > 
> > > > In the geo-rep master log, I see these being triggered:
> > > > 
> > > > # tail -n3
> > > > foo/ssh%3A%2F%2Froot%401.2.3.4%3Agluster%3A%2F%2F127.0.0.1%3Afoo.log
> > > > [2017-03-23 18:33:34.697525] I [master(/gv0/foo):534:crawlwrap]
> > > > _GMaster:
> > > > 20
> > > > crawls, 0 turns
> > > > [2017-03-23 18:34:37.441982] I [master(/gv0/foo):534:crawlwrap]
> > > > _GMaster:
> > > > 20
> > > > crawls, 0 turns
> > > > [2017-03-23 18:35:40.242851] I [master(/gv0/foo):534:crawlwrap]
> > > > _GMaster:
> > > > 20
> > > > crawls, 0 turns
> > > > 
> > > > I don't see any errors in any of the other logs.
> > > > 
> > > > Not sure what else to poke at here. What are the possible values for
> > > > the
> > > > "change_detector" config variable? Would it be worthwhile to test with
> > > > a
> > > > method other than "changelog"? Other thoughts/ideas?
> > > > 
> > > > Thanks in advance!
> > > > 
> > > > j
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > > _______________________________________________
> > > Gluster-users mailing list
> > > Gluster-users at gluster.org
> > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > > 
> 