[Gluster-users] Problem setting up geo-replication: gsyncd using incorrect slave hostname

[Darrell Budic]

I was having trouble setting up geo-replication, and I finally figured out why. Gsyncd is trying to use the wrong (but valid) name for the slave server, and it’s resolving to an address it can’t reach. It does this even though I tried to setup the geo-replication to a specific IP address, and the initial create succeeded.

The environment I’m working with has server1 & server3 with bricks for a replication volume, and my slave server ‘spire’ is located remotely. Because there are multiple paths between the systems, I wanted to force geo-replication to occur with a specific slave address, so I tried to setup replication with an ip based URL. 

Setup/config of the geo-replication works, and appears to start, but is permanently in a failure state:

[root at server1 geo-replication]# gluster volume geo-replication status
No active geo-replication sessions
[root at server1 geo-replication]# gluster volume geo-replication gvOvirt 10.78.4.1::geobk-Ovirt create push-pem
Creating geo-replication session between gvOvirt & 10.78.4.1::geobk-Ovirt has been successful
[root at server1 geo-replication]# gluster volume geo-replication status
 
MASTER NODE                    MASTER VOL    MASTER BRICK             SLAVE                           STATUS         CHECKPOINT STATUS    CRAWL STATUS        
-------------------------------------------------------------------------------------------------------------------------------------------------------
server1.xxx.xxx.xxx    gvOvirt       /v0/ha-engine/gbOvirt    ssh://10.78.4.1::geobk-Ovirt    Not Started    N/A                  N/A                 
server3.xxx.xxx.xxx    gvOvirt       /v0/ha-engine/gbOvirt    ssh://10.78.4.1::geobk-Ovirt    Not Started    N/A                  N/A                 
[root at server1 geo-replication]# gluster volume geo-replication gvOvirt 10.78.4.1::geobk-Ovirt start
Starting geo-replication session between gvOvirt & 10.78.4.1::geobk-Ovirt has been successful
[root at server1 geo-replication]# gluster volume geo-replication status
 
MASTER NODE                    MASTER VOL    MASTER BRICK             SLAVE                           STATUS    CHECKPOINT STATUS    CRAWL STATUS        
--------------------------------------------------------------------------------------------------------------------------------------------------
server1.xxx.xxx.xxx    gvOvirt       /v0/ha-engine/gbOvirt    ssh://10.78.4.1::geobk-Ovirt    faulty    N/A                  N/A                 
server3.xxx.xxx.xxx    gvOvirt       /v0/ha-engine/gbOvirt    ssh://10.78.4.1::geobk-Ovirt    faulty    N/A                  N/A                 

Tracking it down in the logs, it’s because it’s trying to connect to “root at spire” instead of “root at 10.78.4.1”, even though the setup seemed to use 10.78.4.1 just fine:

2014-09-23 14:13:41.595123] I [monitor(monitor):130:monitor] Monitor: starting gsyncd worker
[2014-09-23 14:13:41.764898] I [gsyncd(/v0/ha-engine/gbOvirt):532:main_i] <top>: syncing: gluster://localhost:gvOvirt -> ssh://root@spire:gluster://localhost:geobk-Ovirt
[2014-09-23 14:13:53.40934] E [syncdutils(/v0/ha-engine/gbOvirt):223:log_raise_exception] <top>: connection to peer is broken
[2014-09-23 14:13:53.41244] E [resource(/v0/ha-engine/gbOvirt):204:errlog] Popen: command "ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-gaHXrp/e68410dcff276efa39a94dc5b33e0d8e.sock root at spire /nonexistent/gsyncd --session-owner c9a371cd-8644-4706-a4b7-f12bc2c37ac6 -N --listen --timeout 120 gluster://localhost:geobk-Ovirt" returned with 255, saying:
[2014-09-23 14:13:53.41356] E [resource(/v0/ha-engine/gbOvirt):207:logerr] Popen: ssh> ssh_exchange_identification: Connection closed by remote host
[2014-09-23 14:13:53.41569] I [syncdutils(/v0/ha-engine/gbOvirt):192:finalize] <top>: exiting.
[2014-09-23 14:13:54.44226] I [monitor(monitor):157:monitor] Monitor: worker(/v0/ha-engine/gbOvirt) died in startup phase

So where did it pick this up from? If anything, I think it’d try the full name of the slave server, but it didn’t even try that. The slave’s hostname, btw, is spire.yyy.yyy.xxx, not even in the same domain as the master.

I worked around this by setting up a new hostname resolution for ‘spire' and tweaking the search path so it resolved things the way I wanted (to 10.78.4.1), but that shouldn’t be necessary. Also seems like this might cause security issues (well, ok, probably not since it’s rsyncing over ssh, but traffic could definitely wind up where you didn’t expect it) because it’s not trying to connect to what I explicitly told it to. And in this case, it ran afoul of some sshd root login allowed rules, but could have easily been other firewall rules too. And confusion because the geo-replication status report is misleading, it’s trying to sync to an address that isn’t reflected there.

Seems like a bug to me, but figured I’d check here first.

  -Darrell

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140923/c815a717/attachment.html>