<div dir="ltr"><div>Hi Aravinda and Strahil,</div><div><br></div><div>The cluster is new so it wasn't a big deal to re-do using public addresses. That's done and geo-replication is working.</div><div><br></div><div>Thank you for your help!</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 4 Mar 2020 at 17:17, Aravinda VK <<a href="mailto:aravinda@kadalu.io">aravinda@kadalu.io</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">Hi David,<div><br></div><div>I like the Strahil’s idea of adding remote IPs in /etc/hosts with same name as used in B cluster. Since Geo-replication uses ssh for syncing it should work. Only issue I can think about is if the hostname of cluster B conflicts with hostnames of Cluster A.</div><div><br></div><div>—</div><div>regards</div><div>Aravinda Vishwanathapura</div><div><a href="https://kadalu" target="_blank">https://kadalu</a>.io<br><div><br><blockquote type="cite"><div>On 04-Mar-2020, at 4:13 AM, David Cunningham <<a href="mailto:dcunningham@voisonics.com" target="_blank">dcunningham@voisonics.com</a>> wrote:</div><br><div><div dir="ltr"><div>Hi Strahil,</div><div><br></div><div>The B cluster are communicating with each other via a LAN, and it seems the A cluster has got B's LAN addresses (which aren't accessible from the internet including the A cluster) through the geo-replication process. That being the case, I think we'll have to re-do the B cluster to replicate using public addresses instead of the LAN.</div><div><br></div><div>Thank you.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 3 Mar 2020 at 18:07, Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On March 3, 2020 4:13:38 AM GMT+02:00, David Cunningham <<a href="mailto:dcunningham@voisonics.com" target="_blank">dcunningham@voisonics.com</a>> wrote:<br>
>Hello,<br>
><br>
>Thanks for that. When we re-tried with push-pem from cafs10 (on the<br>
>A/master cluster) it failed with "Unable to mount and fetch slave<br>
>volume<br>
>details." and in the logs we see:<br>
><br>
>[2020-03-03 02:07:42.614911] E<br>
>[name.c:258:af_inet_client_get_remote_sockaddr] 0-gvol0-client-0: DNS<br>
>resolution failed on host nvfs10.local<br>
>[2020-03-03 02:07:42.638824] E<br>
>[name.c:258:af_inet_client_get_remote_sockaddr] 0-gvol0-client-1: DNS<br>
>resolution failed on host nvfs20.local<br>
>[2020-03-03 02:07:42.664493] E<br>
>[name.c:258:af_inet_client_get_remote_sockaddr] 0-gvol0-client-2: DNS<br>
>resolution failed on host nvfs30.local<br>
><br>
>These .local addresses are the LAN addresses that B/slave nodes nvfs10,<br>
>nvfs20, and nvfs30 replicate with. It seems that the A/master needs to<br>
>be<br>
>able to contact those addresses. Is that right? If it is then we'll<br>
>need to<br>
>re-do the B cluster to replicate using publicly accessible IP addresses<br>
>instead of their LAN.<br>
><br>
>Thank you.<br>
><br>
><br>
>On Mon, 2 Mar 2020 at 20:53, Aravinda VK <<a href="mailto:aravinda@kadalu.io" target="_blank">aravinda@kadalu.io</a>> wrote:<br>
><br>
>> Looks like setup issue to me. Copying SSH keys manually is not<br>
>required.<br>
>><br>
>> Command prefix is required while adding to authorized_keys file in<br>
>each<br>
>> remote nodes. That will not be available if ssh keys are added<br>
>manually.<br>
>><br>
>> Geo-rep specifies /nonexisting/gsyncd in the command to make sure it<br>
>> connects via the actual command specified in authorized_keys file, in<br>
>your<br>
>> case Geo-replication is actually looking for gsyncd command in<br>
>> /nonexisting/gsyncd path.<br>
>><br>
>> Please try with push-pem option during Geo-rep create command.<br>
>><br>
>> —<br>
>> regards<br>
>> Aravinda Vishwanathapura<br>
>> <a href="https://kadalu.io/" rel="noreferrer" target="_blank">https://kadalu.io</a><br>
>><br>
>><br>
>> On 02-Mar-2020, at 6:03 AM, David Cunningham<br>
><<a href="mailto:dcunningham@voisonics.com" target="_blank">dcunningham@voisonics.com</a>><br>
>> wrote:<br>
>><br>
>> Hello,<br>
>><br>
>> We've set up geo-replication but it isn't actually syncing. Scenario<br>
>is<br>
>> that we have two GFS clusters. Cluster A has nodes cafs10, cafs20,<br>
>and<br>
>> cafs30, replicating with each other over a LAN. Cluster B has nodes<br>
>nvfs10,<br>
>> nvfs20, and nvfs30 also replicating with each other over a LAN. We<br>
>are<br>
>> geo-replicating data from the A cluster to the B cluster over the<br>
>internet.<br>
>> SSH key access is set up, allowing all the A nodes password-less<br>
>access to<br>
>> root on nvfs10<br>
>><br>
>> Geo-replication was set up using these commands, run on cafs10:<br>
>><br>
>> gluster volume geo-replication gvol0 <a href="http://nvfs10.example.com" target="_blank">nvfs10.example.com</a>::gvol0 create<br>
>> ssh-port 8822 no-verify<br>
>> gluster volume geo-replication gvol0 <a href="http://nvfs10.example.com" target="_blank">nvfs10.example.com</a>::gvol0 config<br>
>> remote-gsyncd /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd<br>
>> gluster volume geo-replication gvol0 <a href="http://nvfs10.example.com" target="_blank">nvfs10.example.com</a>::gvol0 start<br>
>><br>
>> However after a very short period of the status being<br>
>"Initializing..."<br>
>> the status then sits on "Passive":<br>
>><br>
>> # gluster volume geo-replication gvol0 <a href="http://nvfs10.example.com" target="_blank">nvfs10.example.com</a>::gvol0<br>
>status<br>
>> MASTER NODE MASTER VOL MASTER BRICK <br>
>SLAVE<br>
>> USER SLAVE SLAVE NODE STATUS <br>
>CRAWL<br>
>> STATUS LAST_SYNCED<br>
>><br>
>><br>
>------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
>> cafs10 gvol0 /nodirectwritedata/gluster/gvol0 root<br>
>> <a href="http://nvfs10.example.com" target="_blank">nvfs10.example.com</a>::gvol0 nvfs30.local Passive N/A<br>
>> N/A<br>
>> cafs30 gvol0 /nodirectwritedata/gluster/gvol0 root<br>
>> <a href="http://nvfs10.example.com" target="_blank">nvfs10.example.com</a>::gvol0 N/A Created N/A<br>
>> N/A<br>
>> cafs20 gvol0 /nodirectwritedata/gluster/gvol0 root<br>
>> <a href="http://nvfs10.example.com" target="_blank">nvfs10.example.com</a>::gvol0 N/A Created N/A<br>
>> N/A<br>
>><br>
>> So my questions are:<br>
>> 1. Why does the status on cafs10 mention "nvfs30.local"? That's the<br>
>LAN<br>
>> address that nvfs10 replicates with nvfs30 using. It's not accessible<br>
>from<br>
>> the A cluster, and I didn't use it when configuring geo-replication.<br>
>> 2. Why does geo-replication sit in Passive status?<br>
>><br>
>> Thanks very much for any assistance.<br>
>><br>
>><br>
>> On Tue, 25 Feb 2020 at 15:46, David Cunningham<br>
><<a href="mailto:dcunningham@voisonics.com" target="_blank">dcunningham@voisonics.com</a>><br>
>> wrote:<br>
>><br>
>>> Hi Aravinda and Sunny,<br>
>>><br>
>>> Thank you for the replies. We have 3 replicating nodes on the master<br>
>>> side, and want to geo-replicate their data to the remote slave side.<br>
>As I<br>
>>> understand it if the master node which had the geo-replication<br>
>create<br>
>>> command run goes down then another node will take over pushing<br>
>updates to<br>
>>> the remote slave. Is that right?<br>
>>><br>
>>> We have already taken care of adding all master node's SSH keys to<br>
>the<br>
>>> remote slave's authorized_keys externally, so won't include the<br>
>push-pem<br>
>>> part of the create command.<br>
>>><br>
>>> Mostly I wanted to confirm the geo-replication behaviour on the<br>
>>> replicating master nodes if one of them goes down.<br>
>>><br>
>>> Thank you!<br>
>>><br>
>>><br>
>>> On Tue, 25 Feb 2020 at 14:32, Aravinda VK <<a href="mailto:aravinda@kadalu.io" target="_blank">aravinda@kadalu.io</a>><br>
>wrote:<br>
>>><br>
>>>> Hi David,<br>
>>>><br>
>>>><br>
>>>> On 25-Feb-2020, at 3:45 AM, David Cunningham<br>
><<a href="mailto:dcunningham@voisonics.com" target="_blank">dcunningham@voisonics.com</a>><br>
>>>> wrote:<br>
>>>><br>
>>>> Hello,<br>
>>>><br>
>>>> I've a couple of questions on geo-replication that hopefully<br>
>someone can<br>
>>>> help with:<br>
>>>><br>
>>>> 1. If there are multiple nodes in a cluster on the master side<br>
>(pushing<br>
>>>> updates to the geo-replication slave), which node actually does the<br>
>>>> pushing? Does GlusterFS decide itself automatically?<br>
>>>><br>
>>>><br>
>>>> Once Geo-replication session is started, one worker will be started<br>
>>>> corresponding to each Master bricks. Each worker identifies the<br>
>changes<br>
>>>> happened in respective brick and sync those changes via Mount. This<br>
>way<br>
>>>> load is distributed among Master nodes. In case of Replica sub<br>
>volume, one<br>
>>>> worker among the Replica group will become active and participate<br>
>in the<br>
>>>> syncing. Other bricks in that Replica group will remain Passive.<br>
>Passive<br>
>>>> worker will become Active if the previously Active brick goes down<br>
>(This is<br>
>>>> because all Replica bricks will have the same set of changes,<br>
>syncing from<br>
>>>> each worker is redundant).<br>
>>>><br>
>>>><br>
>>>> 2.With regard to copying SSH keys, presumably the SSH key of all<br>
>master<br>
>>>> nodes should be authorized on the geo-replication client side?<br>
>>>><br>
>>>><br>
>>>> Geo-replication session is established between one master node and<br>
>one<br>
>>>> remote node. If Geo-rep create command is successful then,<br>
>>>><br>
>>>> - SSH keys generated in all master nodes<br>
>>>> - Public keys from all master nodes are copied to initiator Master<br>
>node<br>
>>>> - Public keys copied to the Remote node specified in the create<br>
>command<br>
>>>> - Master public keys are distributed to all nodes of remote Cluster<br>
>and<br>
>>>> added to respective ~/.ssh/authorized_keys<br>
>>>><br>
>>>> After successful Geo-rep create command, any Master node can<br>
>connect to<br>
>>>> any remote node via ssh.<br>
>>>><br>
>>>> Security: Command prefix is added while adding public key to remote<br>
>>>> node’s authorized_keys file, So that if anyone gain access using<br>
>this key<br>
>>>> can access only gsyncd command.<br>
>>>><br>
>>>> ```<br>
>>>> command=gsyncd ssh-key….<br>
>>>> ```<br>
>>>><br>
>>>><br>
>>>><br>
>>>> Thanks for your help.<br>
>>>><br>
>>>> --<br>
>>>> David Cunningham, Voisonics Limited<br>
>>>> <a href="http://voisonics.com/" rel="noreferrer" target="_blank">http://voisonics.com/</a><br>
>>>> USA: +1 213 221 1092<br>
>>>> New Zealand: +64 (0)28 2558 3782<br>
>>>> ________<br>
>>>><br>
>>>><br>
>>>><br>
>>>> Community Meeting Calendar:<br>
>>>><br>
>>>> Schedule -<br>
>>>> Every Tuesday at 14:30 IST / 09:00 UTC<br>
>>>> Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
>>>><br>
>>>> Gluster-users mailing list<br>
>>>> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>>>> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
>>>><br>
>>>><br>
>>>><br>
>>>> —<br>
>>>> regards<br>
>>>> Aravinda Vishwanathapura<br>
>>>> <a href="https://kadalu.io/" rel="noreferrer" target="_blank">https://kadalu.io</a><br>
>>>><br>
>>>><br>
>>><br>
>>> --<br>
>>> David Cunningham, Voisonics Limited<br>
>>> <a href="http://voisonics.com/" rel="noreferrer" target="_blank">http://voisonics.com/</a><br>
>>> USA: +1 213 221 1092<br>
>>> New Zealand: +64 (0)28 2558 3782<br>
>>><br>
>><br>
>><br>
>> --<br>
>> David Cunningham, Voisonics Limited<br>
>> <a href="http://voisonics.com/" rel="noreferrer" target="_blank">http://voisonics.com/</a><br>
>> USA: +1 213 221 1092<br>
>> New Zealand: +64 (0)28 2558 3782<br>
>><br>
>><br>
>><br>
<br>
Hey David,<br>
<br>
Why don't you set the B cluster's hostnames in /etc/hosts of all A cluster nodes ?<br>
<br>
Maybe you won't need to rebuild the whole B cluster.<br>
<br>
I guess the A cluster nodes nees to be able to reach all nodes from B cluster, so you might need to change the firewall settings.<br>
<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>David Cunningham, Voisonics Limited<br><a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>USA: +1 213 221 1092<br>New Zealand: +64 (0)28 2558 3782</div></div></div></div></div></div></div></div></div></div></div>
</div></blockquote></div><br></div></div></blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>David Cunningham, Voisonics Limited<br><a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>USA: +1 213 221 1092<br>New Zealand: +64 (0)28 2558 3782</div></div></div></div></div></div></div></div></div></div></div>