[Gluster-users] Geo-replication status always on 'Created'

Maurya M mauryam at gmail.com
Sun Mar 24 11:43:36 UTC 2019 

did all the suggestion as mentioned in the log trace , have another setup
using root user , but there i have issue on the ssh command as i am unable
to change the ssh port to use default 22, but my servers (azure aks
engine)  are configure to using 2222 where i am unable to change the ports
, restart of ssh service giving me error!

Is this syntax correct to config the ssh-command:
gluster volume geo-replication vol_041afbc53746053368a1840607636e97
xxx.xx.xxx.xx::vol_a5aee81a873c043c99a938adcb5b5781 *config ssh-command
'/usr/sbin/sshd -D  -p 2222'*

On Sun, Mar 24, 2019 at 4:38 PM Maurya M <mauryam at gmail.com> wrote:

> Did give the persmission on both "/var/log/glusterfs/" &
> "/var/lib/glusterd/" too, but seems the directory where i mounted using
> heketi is having issues:
>
> [2019-03-22 09:48:21.546308] E [syncdutils(worker
> /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):305:log_raise_exception]
> <top>: connection to peer is broken
>
> [2019-03-22 09:48:21.546662] E [syncdutils(worker
> /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):309:log_raise_exception]
> <top>: getting "No such file or directory"errors is most likely due to
> MISCONFIGURATION, please remove all the public keys added by
> geo-replication from authorized_keys file in slave nodes and run
> Geo-replication create command again.
>
> [2019-03-22 09:48:21.546736] E [syncdutils(worker
> /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):316:log_raise_exception]
> <top>: If `gsec_create container` was used, then run `gluster volume
> geo-replication <MASTERVOL> [<SLAVEUSER>@]<SLAVEHOST>::<SLAVEVOL> config
> remote-gsyncd <GSYNCD_PATH> (Example GSYNCD_PATH:
> `/usr/libexec/glusterfs/gsyncd`)
>
> [2019-03-22 09:48:21.546858] E [syncdutils(worker
> /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):801:errlog]
> Popen: command returned error    cmd=ssh -oPasswordAuthentication=no
> -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem
> -p 22 -oControlMaster=auto -S
> /tmp/gsyncd-aux-ssh-OaPGc3/c784230c9648efa4d529975bd779c551.sock
> azureuser at 172.16.201.35 /nonexistent/gsyncd slave
> vol_041afbc53746053368a1840607636e97
> azureuser at 172.16.201.35::vol_a5aee81a873c043c99a938adcb5b5781
> --master-node 172.16.189.4 --master-node-id
> dd4efc35-4b86-4901-9c00-483032614c35 --master-brick
> /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick
> --local-node 172.16.201.35 --local-node-id
> 7eb0a2b6-c4d6-41b1-a346-0638dbf8d779 --slave-timeout 120 --slave-log-level
> INFO --slave-gluster-log-level INFO --slave-gluster-command-dir
> /usr/sbin      error=127
>
> [2019-03-22 09:48:21.546977] E [syncdutils(worker
> /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):805:logerr]
> Popen: ssh> bash: /nonexistent/gsyncd: No such file or directory
>
> [2019-03-22 09:48:21.565583] I [repce(agent
> /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):80:service_loop]
> RepceServer: terminating on reaching EOF.
>
> [2019-03-22 09:48:21.565745] I [monitor(monitor):266:monitor] Monitor:
> worker died before establishing connection
> brick=/var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick
>
> [2019-03-22 09:48:21.579195] I
> [gsyncdstatus(monitor):245:set_worker_status] GeorepStatus: Worker Status
> Change status=Faulty
>
> On Fri, Mar 22, 2019 at 10:23 PM Sunny Kumar <sunkumar at redhat.com> wrote:
>
>> Hi Maurya,
>>
>> Looks like hook script is failed to set permissions for azureuser on
>> "/var/log/glusterfs".
>> You can assign permission manually for directory then it will work.
>>
>> -Sunny
>>
>> On Fri, Mar 22, 2019 at 2:07 PM Maurya M <mauryam at gmail.com> wrote:
>> >
>> > hi Sunny,
>> >  Passwordless ssh to :
>> >
>> > ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
>> /var/lib/glusterd/geo-replication/secret.pem -p 22
>> azureuser at 172.16.201.35
>> >
>> > is login, but when the whole command is run getting permission issues
>> again::
>> >
>> > ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
>> /var/lib/glusterd/geo-replication/secret.pem -p 22
>> azureuser at 172.16.201.35 gluster --xml --remote-host=localhost volume
>> info vol_a5aee81a873c043c99a938adcb5b5781 -v
>> > ERROR: failed to create logfile "/var/log/glusterfs/cli.log"
>> (Permission denied)
>> > ERROR: failed to open logfile /var/log/glusterfs/cli.log
>> >
>> > any idea here ?
>> >
>> > thanks,
>> > Maurya
>> >
>> >
>> > On Thu, Mar 21, 2019 at 2:43 PM Maurya M <mauryam at gmail.com> wrote:
>> >>
>> >> hi Sunny,
>> >>  i did use the [1] link for the setup, when i encountered this error
>> during ssh-copy-id : (so setup the passwordless ssh, by manually copied the
>> private/ public keys to all the nodes , both master & slave)
>> >>
>> >> [root at k8s-agentpool1-24779565-1 ~]# ssh-copy-id geouser at xxx.xx.xxx.x
>> >> /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed:
>> "/root/.ssh/id_rsa.pub"
>> >> The authenticity of host ' xxx.xx.xxx.x   ( xxx.xx.xxx.x  )' can't be
>> established.
>> >> ECDSA key fingerprint is
>> SHA256:B2rNaocIcPjRga13oTnopbJ5KjI/7l5fMANXc+KhA9s.
>> >> ECDSA key fingerprint is
>> MD5:1b:70:f9:7a:bf:35:33:47:0c:f2:c1:cd:21:e2:d3:75.
>> >> Are you sure you want to continue connecting (yes/no)? yes
>> >> /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s),
>> to filter out any that are already installed
>> >> /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you
>> are prompted now it is to install the new keys
>> >> Permission denied (publickey).
>> >>
>> >> To start afresh what all needs to teardown / delete, do we have any
>> script for it ? where all the pem keys do i need to delete?
>> >>
>> >> thanks,
>> >> Maurya
>> >>
>> >> On Thu, Mar 21, 2019 at 2:12 PM Sunny Kumar <sunkumar at redhat.com>
>> wrote:
>> >>>
>> >>> Hey you can start a fresh I think you are not following proper setup
>> steps.
>> >>>
>> >>> Please follow these steps [1] to create geo-rep session, you can
>> >>> delete the old one and do a fresh start. Or alternative you can use
>> >>> this tool[2] to setup geo-rep.
>> >>>
>> >>>
>> >>> [1].
>> https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/
>> >>> [2]. http://aravindavk.in/blog/gluster-georep-tools/
>> >>>
>> >>>
>> >>> /Sunny
>> >>>
>> >>> On Thu, Mar 21, 2019 at 11:28 AM Maurya M <mauryam at gmail.com> wrote:
>> >>> >
>> >>> > Hi Sunil,
>> >>> >  I did run the on the slave node :
>> >>> >  /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh azureuser
>> vol_041afbc53746053368a1840607636e97 vol_a5aee81a873c043c99a938adcb5b5781
>> >>> > getting this message "/home/azureuser/common_secret.pem.pub not
>> present. Please run geo-replication command on master with push-pem option
>> to generate the file"
>> >>> >
>> >>> > So went back and created the session again, no change, so manually
>> copied the common_secret.pem.pub to /home/azureuser/ but still the
>> set_geo_rep_pem_keys.sh is looking the pem file in different name :
>> COMMON_SECRET_PEM_PUB=${master_vol}_${slave_vol}_common_secret.pem.pub ,
>> change the name of pem , ran the command again :
>> >>> >
>> >>> >  /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh azureuser
>> vol_041afbc53746053368a1840607636e97 vol_a5aee81a873c043c99a938adcb5b5781
>> >>> > Successfully copied file.
>> >>> > Command executed successfully.
>> >>> >
>> >>> >
>> >>> > - went back and created the session , start the geo-replication ,
>> still seeing the  same error in logs. Any ideas ?
>> >>> >
>> >>> > thanks,
>> >>> > Maurya
>> >>> >
>> >>> >
>> >>> >
>> >>> > On Wed, Mar 20, 2019 at 11:07 PM Sunny Kumar <sunkumar at redhat.com>
>> wrote:
>> >>> >>
>> >>> >> Hi Maurya,
>> >>> >>
>> >>> >> I guess you missed last trick to distribute keys in slave node. I
>> see
>> >>> >> this is non-root geo-rep setup so please try this:
>> >>> >>
>> >>> >>
>> >>> >> Run the following command as root in any one of Slave node.
>> >>> >>
>> >>> >> /usr/local/libexec/glusterfs/set_geo_rep_pem_keys.sh  <slave_user>
>> >>> >> <master_volume> <slave_volume>
>> >>> >>
>> >>> >> - Sunny
>> >>> >>
>> >>> >> On Wed, Mar 20, 2019 at 10:47 PM Maurya M <mauryam at gmail.com>
>> wrote:
>> >>> >> >
>> >>> >> > Hi all,
>> >>> >> >  Have setup a 3 master nodes - 3 slave nodes (gluster 4.1) for
>> geo-replication, but once have the geo-replication configure the status is
>> always on "Created',
>> >>> >> > even after have force start the session.
>> >>> >> >
>> >>> >> > On close inspect of the logs on the master node seeing this
>> error:
>> >>> >> >
>> >>> >> > "E [syncdutils(monitor):801:errlog] Popen: command returned
>> error   cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
>> /var/lib/glusterd/geo-replication/secret.pem -p 22 azureuser at xxxxx.xxxx..xxx.
>> gluster --xml --remote-host=localhost volume info
>> vol_a5ae34341a873c043c99a938adcb5b5781      error=255"
>> >>> >> >
>> >>> >> > Any ideas what is issue?
>> >>> >> >
>> >>> >> > thanks,
>> >>> >> > Maurya
>> >>> >> >
>> >>> >> > _______________________________________________
>> >>> >> > Gluster-users mailing list
>> >>> >> > Gluster-users at gluster.org
>> >>> >> > https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190324/40e5eef1/attachment.html>

More information about the Gluster-users mailing list