<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi David,<div class=""><br class=""></div><div class="">I like the Strahil’s idea of adding remote IPs in /etc/hosts with same name as used in B cluster. Since Geo-replication uses ssh for syncing it should work. Only issue I can think about is if the hostname of cluster B conflicts with hostnames of Cluster A.</div><div class=""><br class=""></div><div class="">—</div><div class="">regards</div><div class="">Aravinda Vishwanathapura</div><div class=""><a href="https://kadalu" class="">https://kadalu</a>.io<br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 04-Mar-2020, at 4:13 AM, David Cunningham <<a href="mailto:dcunningham@voisonics.com" class="">dcunningham@voisonics.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">Hi Strahil,</div><div class=""><br class=""></div><div class="">The B cluster are communicating with each other via a LAN, and it seems the A cluster has got B's LAN addresses (which aren't accessible from the internet including the A cluster) through the geo-replication process. That being the case, I think we'll have to re-do the B cluster to replicate using public addresses instead of the LAN.</div><div class=""><br class=""></div><div class="">Thank you.</div><div class=""><br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 3 Mar 2020 at 18:07, Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" class="">hunter86_bg@yahoo.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On March 3, 2020 4:13:38 AM GMT+02:00, David Cunningham <<a href="mailto:dcunningham@voisonics.com" target="_blank" class="">dcunningham@voisonics.com</a>> wrote:<br class="">
>Hello,<br class="">
><br class="">
>Thanks for that. When we re-tried with push-pem from cafs10 (on the<br class="">
>A/master cluster) it failed with "Unable to mount and fetch slave<br class="">
>volume<br class="">
>details." and in the logs we see:<br class="">
><br class="">
>[2020-03-03 02:07:42.614911] E<br class="">
>[name.c:258:af_inet_client_get_remote_sockaddr] 0-gvol0-client-0: DNS<br class="">
>resolution failed on host nvfs10.local<br class="">
>[2020-03-03 02:07:42.638824] E<br class="">
>[name.c:258:af_inet_client_get_remote_sockaddr] 0-gvol0-client-1: DNS<br class="">
>resolution failed on host nvfs20.local<br class="">
>[2020-03-03 02:07:42.664493] E<br class="">
>[name.c:258:af_inet_client_get_remote_sockaddr] 0-gvol0-client-2: DNS<br class="">
>resolution failed on host nvfs30.local<br class="">
><br class="">
>These .local addresses are the LAN addresses that B/slave nodes nvfs10,<br class="">
>nvfs20, and nvfs30 replicate with. It seems that the A/master needs to<br class="">
>be<br class="">
>able to contact those addresses. Is that right? If it is then we'll<br class="">
>need to<br class="">
>re-do the B cluster to replicate using publicly accessible IP addresses<br class="">
>instead of their LAN.<br class="">
><br class="">
>Thank you.<br class="">
><br class="">
><br class="">
>On Mon, 2 Mar 2020 at 20:53, Aravinda VK <<a href="mailto:aravinda@kadalu.io" target="_blank" class="">aravinda@kadalu.io</a>> wrote:<br class="">
><br class="">
>> Looks like setup issue to me. Copying SSH keys manually is not<br class="">
>required.<br class="">
>><br class="">
>> Command prefix is required while adding to authorized_keys file in<br class="">
>each<br class="">
>> remote nodes. That will not be available if ssh keys are added<br class="">
>manually.<br class="">
>><br class="">
>> Geo-rep specifies /nonexisting/gsyncd in the command to make sure it<br class="">
>> connects via the actual command specified in authorized_keys file, in<br class="">
>your<br class="">
>> case Geo-replication is actually looking for gsyncd command in<br class="">
>> /nonexisting/gsyncd path.<br class="">
>><br class="">
>> Please try with push-pem option during Geo-rep create command.<br class="">
>><br class="">
>> —<br class="">
>> regards<br class="">
>> Aravinda Vishwanathapura<br class="">
>> <a href="https://kadalu.io/" rel="noreferrer" target="_blank" class="">https://kadalu.io</a><br class="">
>><br class="">
>><br class="">
>> On 02-Mar-2020, at 6:03 AM, David Cunningham<br class="">
><<a href="mailto:dcunningham@voisonics.com" target="_blank" class="">dcunningham@voisonics.com</a>><br class="">
>> wrote:<br class="">
>><br class="">
>> Hello,<br class="">
>><br class="">
>> We've set up geo-replication but it isn't actually syncing. Scenario<br class="">
>is<br class="">
>> that we have two GFS clusters. Cluster A has nodes cafs10, cafs20,<br class="">
>and<br class="">
>> cafs30, replicating with each other over a LAN. Cluster B has nodes<br class="">
>nvfs10,<br class="">
>> nvfs20, and nvfs30 also replicating with each other over a LAN. We<br class="">
>are<br class="">
>> geo-replicating data from the A cluster to the B cluster over the<br class="">
>internet.<br class="">
>> SSH key access is set up, allowing all the A nodes password-less<br class="">
>access to<br class="">
>> root on nvfs10<br class="">
>><br class="">
>> Geo-replication was set up using these commands, run on cafs10:<br class="">
>><br class="">
>> gluster volume geo-replication gvol0 <a href="http://nvfs10.example.com" class="">nvfs10.example.com</a>::gvol0 create<br class="">
>> ssh-port 8822 no-verify<br class="">
>> gluster volume geo-replication gvol0 <a href="http://nvfs10.example.com" class="">nvfs10.example.com</a>::gvol0 config<br class="">
>> remote-gsyncd /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd<br class="">
>> gluster volume geo-replication gvol0 <a href="http://nvfs10.example.com" class="">nvfs10.example.com</a>::gvol0 start<br class="">
>><br class="">
>> However after a very short period of the status being<br class="">
>"Initializing..."<br class="">
>> the status then sits on "Passive":<br class="">
>><br class="">
>> # gluster volume geo-replication gvol0 <a href="http://nvfs10.example.com" class="">nvfs10.example.com</a>::gvol0<br class="">
>status<br class="">
>> MASTER NODE MASTER VOL MASTER BRICK <br class="">
>SLAVE<br class="">
>> USER SLAVE SLAVE NODE STATUS <br class="">
>CRAWL<br class="">
>> STATUS LAST_SYNCED<br class="">
>><br class="">
>><br class="">
>------------------------------------------------------------------------------------------------------------------------------------------------------------------<br class="">
>> cafs10 gvol0 /nodirectwritedata/gluster/gvol0 root<br class="">
>> <a href="http://nvfs10.example.com" class="">nvfs10.example.com</a>::gvol0 nvfs30.local Passive N/A<br class="">
>> N/A<br class="">
>> cafs30 gvol0 /nodirectwritedata/gluster/gvol0 root<br class="">
>> <a href="http://nvfs10.example.com" class="">nvfs10.example.com</a>::gvol0 N/A Created N/A<br class="">
>> N/A<br class="">
>> cafs20 gvol0 /nodirectwritedata/gluster/gvol0 root<br class="">
>> <a href="http://nvfs10.example.com" class="">nvfs10.example.com</a>::gvol0 N/A Created N/A<br class="">
>> N/A<br class="">
>><br class="">
>> So my questions are:<br class="">
>> 1. Why does the status on cafs10 mention "nvfs30.local"? That's the<br class="">
>LAN<br class="">
>> address that nvfs10 replicates with nvfs30 using. It's not accessible<br class="">
>from<br class="">
>> the A cluster, and I didn't use it when configuring geo-replication.<br class="">
>> 2. Why does geo-replication sit in Passive status?<br class="">
>><br class="">
>> Thanks very much for any assistance.<br class="">
>><br class="">
>><br class="">
>> On Tue, 25 Feb 2020 at 15:46, David Cunningham<br class="">
><<a href="mailto:dcunningham@voisonics.com" target="_blank" class="">dcunningham@voisonics.com</a>><br class="">
>> wrote:<br class="">
>><br class="">
>>> Hi Aravinda and Sunny,<br class="">
>>><br class="">
>>> Thank you for the replies. We have 3 replicating nodes on the master<br class="">
>>> side, and want to geo-replicate their data to the remote slave side.<br class="">
>As I<br class="">
>>> understand it if the master node which had the geo-replication<br class="">
>create<br class="">
>>> command run goes down then another node will take over pushing<br class="">
>updates to<br class="">
>>> the remote slave. Is that right?<br class="">
>>><br class="">
>>> We have already taken care of adding all master node's SSH keys to<br class="">
>the<br class="">
>>> remote slave's authorized_keys externally, so won't include the<br class="">
>push-pem<br class="">
>>> part of the create command.<br class="">
>>><br class="">
>>> Mostly I wanted to confirm the geo-replication behaviour on the<br class="">
>>> replicating master nodes if one of them goes down.<br class="">
>>><br class="">
>>> Thank you!<br class="">
>>><br class="">
>>><br class="">
>>> On Tue, 25 Feb 2020 at 14:32, Aravinda VK <<a href="mailto:aravinda@kadalu.io" target="_blank" class="">aravinda@kadalu.io</a>><br class="">
>wrote:<br class="">
>>><br class="">
>>>> Hi David,<br class="">
>>>><br class="">
>>>><br class="">
>>>> On 25-Feb-2020, at 3:45 AM, David Cunningham<br class="">
><<a href="mailto:dcunningham@voisonics.com" target="_blank" class="">dcunningham@voisonics.com</a>><br class="">
>>>> wrote:<br class="">
>>>><br class="">
>>>> Hello,<br class="">
>>>><br class="">
>>>> I've a couple of questions on geo-replication that hopefully<br class="">
>someone can<br class="">
>>>> help with:<br class="">
>>>><br class="">
>>>> 1. If there are multiple nodes in a cluster on the master side<br class="">
>(pushing<br class="">
>>>> updates to the geo-replication slave), which node actually does the<br class="">
>>>> pushing? Does GlusterFS decide itself automatically?<br class="">
>>>><br class="">
>>>><br class="">
>>>> Once Geo-replication session is started, one worker will be started<br class="">
>>>> corresponding to each Master bricks. Each worker identifies the<br class="">
>changes<br class="">
>>>> happened in respective brick and sync those changes via Mount. This<br class="">
>way<br class="">
>>>> load is distributed among Master nodes. In case of Replica sub<br class="">
>volume, one<br class="">
>>>> worker among the Replica group will become active and participate<br class="">
>in the<br class="">
>>>> syncing. Other bricks in that Replica group will remain Passive.<br class="">
>Passive<br class="">
>>>> worker will become Active if the previously Active brick goes down<br class="">
>(This is<br class="">
>>>> because all Replica bricks will have the same set of changes,<br class="">
>syncing from<br class="">
>>>> each worker is redundant).<br class="">
>>>><br class="">
>>>><br class="">
>>>> 2.With regard to copying SSH keys, presumably the SSH key of all<br class="">
>master<br class="">
>>>> nodes should be authorized on the geo-replication client side?<br class="">
>>>><br class="">
>>>><br class="">
>>>> Geo-replication session is established between one master node and<br class="">
>one<br class="">
>>>> remote node. If Geo-rep create command is successful then,<br class="">
>>>><br class="">
>>>> - SSH keys generated in all master nodes<br class="">
>>>> - Public keys from all master nodes are copied to initiator Master<br class="">
>node<br class="">
>>>> - Public keys copied to the Remote node specified in the create<br class="">
>command<br class="">
>>>> - Master public keys are distributed to all nodes of remote Cluster<br class="">
>and<br class="">
>>>> added to respective ~/.ssh/authorized_keys<br class="">
>>>><br class="">
>>>> After successful Geo-rep create command, any Master node can<br class="">
>connect to<br class="">
>>>> any remote node via ssh.<br class="">
>>>><br class="">
>>>> Security: Command prefix is added while adding public key to remote<br class="">
>>>> node’s authorized_keys file, So that if anyone gain access using<br class="">
>this key<br class="">
>>>> can access only gsyncd command.<br class="">
>>>><br class="">
>>>> ```<br class="">
>>>> command=gsyncd ssh-key….<br class="">
>>>> ```<br class="">
>>>><br class="">
>>>><br class="">
>>>><br class="">
>>>> Thanks for your help.<br class="">
>>>><br class="">
>>>> --<br class="">
>>>> David Cunningham, Voisonics Limited<br class="">
>>>> <a href="http://voisonics.com/" rel="noreferrer" target="_blank" class="">http://voisonics.com/</a><br class="">
>>>> USA: +1 213 221 1092<br class="">
>>>> New Zealand: +64 (0)28 2558 3782<br class="">
>>>> ________<br class="">
>>>><br class="">
>>>><br class="">
>>>><br class="">
>>>> Community Meeting Calendar:<br class="">
>>>><br class="">
>>>> Schedule -<br class="">
>>>> Every Tuesday at 14:30 IST / 09:00 UTC<br class="">
>>>> Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank" class="">https://bluejeans.com/441850968</a><br class="">
>>>><br class="">
>>>> Gluster-users mailing list<br class="">
>>>> <a href="mailto:Gluster-users@gluster.org" target="_blank" class="">Gluster-users@gluster.org</a><br class="">
>>>> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br class="">
>>>><br class="">
>>>><br class="">
>>>><br class="">
>>>> —<br class="">
>>>> regards<br class="">
>>>> Aravinda Vishwanathapura<br class="">
>>>> <a href="https://kadalu.io/" rel="noreferrer" target="_blank" class="">https://kadalu.io</a><br class="">
>>>><br class="">
>>>><br class="">
>>><br class="">
>>> --<br class="">
>>> David Cunningham, Voisonics Limited<br class="">
>>> <a href="http://voisonics.com/" rel="noreferrer" target="_blank" class="">http://voisonics.com/</a><br class="">
>>> USA: +1 213 221 1092<br class="">
>>> New Zealand: +64 (0)28 2558 3782<br class="">
>>><br class="">
>><br class="">
>><br class="">
>> --<br class="">
>> David Cunningham, Voisonics Limited<br class="">
>> <a href="http://voisonics.com/" rel="noreferrer" target="_blank" class="">http://voisonics.com/</a><br class="">
>> USA: +1 213 221 1092<br class="">
>> New Zealand: +64 (0)28 2558 3782<br class="">
>><br class="">
>><br class="">
>><br class="">
<br class="">
Hey David,<br class="">
<br class="">
Why don't you set the B cluster's hostnames in /etc/hosts of all A cluster nodes ?<br class="">
<br class="">
Maybe you won't need to rebuild the whole B cluster.<br class="">
<br class="">
I guess the A cluster nodes nees to be able to reach all nodes from B cluster, so you might need to change the firewall settings.<br class="">
<br class="">
<br class="">
Best Regards,<br class="">
Strahil Nikolov<br class="">
</blockquote></div><br clear="all" class=""><br class="">-- <br class=""><div dir="ltr" class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class="">David Cunningham, Voisonics Limited<br class=""><a href="http://voisonics.com/" target="_blank" class="">http://voisonics.com/</a><br class="">USA: +1 213 221 1092<br class="">New Zealand: +64 (0)28 2558 3782</div></div></div></div></div></div></div></div></div></div></div>
</div></blockquote></div><br class=""></div></body></html>