[Gluster-users] Geo-replication

Mon Mar 2 06:56:29 UTC 2020

On March 2, 2020 2:33:10 AM GMT+02:00, David Cunningham <dcunningham at voisonics.com> wrote:
>Hello,
>
>We've set up geo-replication but it isn't actually syncing. Scenario is
>that we have two GFS clusters. Cluster A has nodes cafs10, cafs20, and
>cafs30, replicating with each other over a LAN. Cluster B has nodes
>nvfs10,
>nvfs20, and nvfs30 also replicating with each other over a LAN. We are
>geo-replicating data from the A cluster to the B cluster over the
>internet.
>SSH key access is set up, allowing all the A nodes password-less access
>to
>root on nvfs10
>
>Geo-replication was set up using these commands, run on cafs10:
>
>gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 create
>ssh-port 8822 no-verify
>gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 config
>remote-gsyncd /usr/lib/x86_64-linux-gnu/glusterfs/gsyncd
>gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 start
>
>However after a very short period of the status being "Initializing..."
>the
>status then sits on "Passive":
>
># gluster volume geo-replication gvol0 nvfs10.example.com::gvol0 status
>MASTER NODE    MASTER VOL    MASTER BRICK                        SLAVE
>USER
>  SLAVE                         SLAVE NODE      STATUS     CRAWL STATUS
> LAST_SYNCED
>------------------------------------------------------------------------------------------------------------------------------------------------------------------
>cafs10         gvol0         /nodirectwritedata/gluster/gvol0    root
>   nvfs10.example.com::gvol0    nvfs30.local    Passive    N/A
>N/A
>cafs30         gvol0         /nodirectwritedata/gluster/gvol0    root
>   nvfs10.example.com::gvol0    N/A             Created    N/A
>N/A
>cafs20         gvol0         /nodirectwritedata/gluster/gvol0    root
>   nvfs10.example.com::gvol0    N/A             Created    N/A
>N/A
>
>So my questions are:
>1. Why does the status on cafs10 mention "nvfs30.local"? That's the LAN
>address that nvfs10 replicates with nvfs30 using. It's not accessible
>from
>the A cluster, and I didn't use it when configuring geo-replication.
>2. Why does geo-replication sit in Passive status?
>
>Thanks very much for any assistance.
>
>
>On Tue, 25 Feb 2020 at 15:46, David Cunningham
><dcunningham at voisonics.com>
>wrote:
>
>> Hi Aravinda and Sunny,
>>
>> Thank you for the replies. We have 3 replicating nodes on the master
>side,
>> and want to geo-replicate their data to the remote slave side. As I
>> understand it if the master node which had the geo-replication create
>> command run goes down then another node will take over pushing
>updates to
>> the remote slave. Is that right?
>>
>> We have already taken care of adding all master node's SSH keys to
>the
>> remote slave's authorized_keys externally, so won't include the
>push-pem
>> part of the create command.
>>
>> Mostly I wanted to confirm the geo-replication behaviour on the
>> replicating master nodes if one of them goes down.
>>
>> Thank you!
>>
>>
>> On Tue, 25 Feb 2020 at 14:32, Aravinda VK <aravinda at kadalu.io> wrote:
>>
>>> Hi David,
>>>
>>>
>>> On 25-Feb-2020, at 3:45 AM, David Cunningham
><dcunningham at voisonics.com>
>>> wrote:
>>>
>>> Hello,
>>>
>>> I've a couple of questions on geo-replication that hopefully someone
>can
>>> help with:
>>>
>>> 1. If there are multiple nodes in a cluster on the master side
>(pushing
>>> updates to the geo-replication slave), which node actually does the
>>> pushing? Does GlusterFS decide itself automatically?
>>>
>>>
>>> Once Geo-replication session is started, one worker will be started
>>> corresponding to each Master bricks. Each worker identifies the
>changes
>>> happened in respective brick and sync those changes via Mount. This
>way
>>> load is distributed among Master nodes. In case of Replica sub
>volume, one
>>> worker among the Replica group will become active and participate in
>the
>>> syncing. Other bricks in that Replica group will remain Passive.
>Passive
>>> worker will become Active if the previously Active brick goes down
>(This is
>>> because all Replica bricks will have the same set of changes,
>syncing from
>>> each worker is redundant).
>>>
>>>
>>> 2.With regard to copying SSH keys, presumably the SSH key of all
>master
>>> nodes should be authorized on the geo-replication client side?
>>>
>>>
>>> Geo-replication session is established between one master node and
>one
>>> remote node. If Geo-rep create command is successful then,
>>>
>>> - SSH keys generated in all master nodes
>>> - Public keys from all master nodes are copied to initiator Master
>node
>>> - Public keys copied to the Remote node specified in the create
>command
>>> - Master public keys are distributed to all nodes of remote Cluster
>and
>>> added to respective ~/.ssh/authorized_keys
>>>
>>> After successful Geo-rep create command, any Master node can connect
>to
>>> any remote node via ssh.
>>>
>>> Security: Command prefix is added while adding public key to remote
>>> node’s authorized_keys file, So that if anyone gain access using
>this key
>>> can access only gsyncd command.
>>>
>>> ```
>>> command=gsyncd ssh-key….
>>> ```
>>>
>>>
>>>
>>> Thanks for your help.
>>>
>>> --
>>> David Cunningham, Voisonics Limited
>>> http://voisonics.com/
>>> USA: +1 213 221 1092
>>> New Zealand: +64 (0)28 2558 3782
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>> —
>>> regards
>>> Aravinda Vishwanathapura
>>> https://kadalu.io
>>>
>>>
>>
>> --
>> David Cunningham, Voisonics Limited
>> http://voisonics.com/
>> USA: +1 213 221 1092
>> New Zealand: +64 (0)28 2558 3782
>>

Hey David,

What are the logs saying ?
Also, doublecheck access from the master to the slave node via ssh.

Best Regards,
Strahil Nikolov