<div dir="ltr"><div dir="ltr">did all the suggestion as mentioned in the log trace , have another setup using root user , but there i have issue on the ssh command as i am unable to change the ssh port to use default 22, but my servers (azure aks engine) are configure to using 2222 where i am unable to change the ports , restart of ssh service giving me error!<div><br></div><div>Is this syntax correct to config the ssh-command:</div><div>gluster volume geo-replication vol_041afbc53746053368a1840607636e97 xxx.xx.xxx.xx::vol_a5aee81a873c043c99a938adcb5b5781 <b>config ssh-command '/usr/sbin/sshd -D -p 2222'</b><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Mar 24, 2019 at 4:38 PM Maurya M <<a href="mailto:mauryam@gmail.com">mauryam@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Did give the persmission on both "/var/log/glusterfs/" & "/var/lib/glusterd/" too, but seems the directory where i mounted using heketi is having issues:<div><br></div><div><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(31,73,125)">[2019-03-22 09:48:21.546308] E
[syncdutils(worker
/var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):305:log_raise_exception]
<top>: connection to peer is broken</span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(31,73,125);background:yellow">[2019-03-22 09:48:21.546662] E [syncdutils(worker
/var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):309:log_raise_exception]
<top>: getting "No such file or directory"errors is most likely
due to MISCONFIGURATION, please remove all the public keys added by
geo-replication from authorized_keys file in slave nodes and run
Geo-replication create command again.</span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(31,73,125);background:yellow">[2019-03-22 09:48:21.546736] E [syncdutils(worker
/var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):316:log_raise_exception]
<top>: If `gsec_create container` was used, then run `gluster volume
geo-replication <MASTERVOL>
[<SLAVEUSER>@]<SLAVEHOST>::<SLAVEVOL> config remote-gsyncd
<GSYNCD_PATH> (Example GSYNCD_PATH: `/usr/libexec/glusterfs/gsyncd`)</span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(31,73,125);background:yellow">[2019-03-22 09:48:21.546858] E [syncdutils(worker
/var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):801:errlog]
Popen: command returned error cmd=ssh
-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-OaPGc3/c784230c9648efa4d529975bd779c551.sock <a href="mailto:azureuser@172.16.201.35" style="color:blue" target="_blank">azureuser@172.16.201.35</a>
/nonexistent/gsyncd slave vol_041afbc53746053368a1840607636e97 <a href="mailto:azureuser@172.16.201.35::vol_a5aee81a873c043c99a938adcb5b5781" style="color:blue" target="_blank">azureuser@172.16.201.35::vol_a5aee81a873c043c99a938adcb5b5781</a>
--master-node 172.16.189.4 --master-node-id
dd4efc35-4b86-4901-9c00-483032614c35 --master-brick
/var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick
--local-node 172.16.201.35 --local-node-id 7eb0a2b6-c4d6-41b1-a346-0638dbf8d779
--slave-timeout 120 --slave-log-level INFO --slave-gluster-log-level INFO
--slave-gluster-command-dir /usr/sbin error=127</span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(31,73,125);background:yellow">[2019-03-22 09:48:21.546977] E [syncdutils(worker /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):805:logerr]
Popen: ssh> bash: /nonexistent/gsyncd: No such file or directory</span><span style="color:rgb(31,73,125)"></span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(31,73,125)">[2019-03-22 09:48:21.565583] I
[repce(agent
/var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick):80:service_loop]
RepceServer: terminating on reaching EOF.</span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(31,73,125)">[2019-03-22 09:48:21.565745] I
[monitor(monitor):266:monitor] Monitor: worker died before establishing
connection
brick=/var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_b3eab2394433f02f5617012d4ae3c28f/brick</span></p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"><span style="color:rgb(31,73,125)">[2019-03-22 09:48:21.579195] I
[gsyncdstatus(monitor):245:set_worker_status] GeorepStatus: Worker Status
Change status=Faulty</span></p></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 22, 2019 at 10:23 PM Sunny Kumar <<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Maurya,<br>
<br>
Looks like hook script is failed to set permissions for azureuser on<br>
"/var/log/glusterfs".<br>
You can assign permission manually for directory then it will work.<br>
<br>
-Sunny<br>
<br>
On Fri, Mar 22, 2019 at 2:07 PM Maurya M <<a href="mailto:mauryam@gmail.com" target="_blank">mauryam@gmail.com</a>> wrote:<br>
><br>
> hi Sunny,<br>
> Passwordless ssh to :<br>
><br>
> ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 <a href="mailto:azureuser@172.16.201.35" target="_blank">azureuser@172.16.201.35</a><br>
><br>
> is login, but when the whole command is run getting permission issues again::<br>
><br>
> ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 <a href="mailto:azureuser@172.16.201.35" target="_blank">azureuser@172.16.201.35</a> gluster --xml --remote-host=localhost volume info vol_a5aee81a873c043c99a938adcb5b5781 -v<br>
> ERROR: failed to create logfile "/var/log/glusterfs/cli.log" (Permission denied)<br>
> ERROR: failed to open logfile /var/log/glusterfs/cli.log<br>
><br>
> any idea here ?<br>
><br>
> thanks,<br>
> Maurya<br>
><br>
><br>
> On Thu, Mar 21, 2019 at 2:43 PM Maurya M <<a href="mailto:mauryam@gmail.com" target="_blank">mauryam@gmail.com</a>> wrote:<br>
>><br>
>> hi Sunny,<br>
>> i did use the [1] link for the setup, when i encountered this error during ssh-copy-id : (so setup the passwordless ssh, by manually copied the private/ public keys to all the nodes , both master & slave)<br>
>><br>
>> [root@k8s-agentpool1-24779565-1 ~]# ssh-copy-id geouser@xxx.xx.xxx.x<br>
>> /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"<br>
>> The authenticity of host ' xxx.xx.xxx.x ( xxx.xx.xxx.x )' can't be established.<br>
>> ECDSA key fingerprint is SHA256:B2rNaocIcPjRga13oTnopbJ5KjI/7l5fMANXc+KhA9s.<br>
>> ECDSA key fingerprint is MD5:1b:70:f9:7a:bf:35:33:47:0c:f2:c1:cd:21:e2:d3:75.<br>
>> Are you sure you want to continue connecting (yes/no)? yes<br>
>> /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed<br>
>> /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys<br>
>> Permission denied (publickey).<br>
>><br>
>> To start afresh what all needs to teardown / delete, do we have any script for it ? where all the pem keys do i need to delete?<br>
>><br>
>> thanks,<br>
>> Maurya<br>
>><br>
>> On Thu, Mar 21, 2019 at 2:12 PM Sunny Kumar <<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>> wrote:<br>
>>><br>
>>> Hey you can start a fresh I think you are not following proper setup steps.<br>
>>><br>
>>> Please follow these steps [1] to create geo-rep session, you can<br>
>>> delete the old one and do a fresh start. Or alternative you can use<br>
>>> this tool[2] to setup geo-rep.<br>
>>><br>
>>><br>
>>> [1]. <a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/" rel="noreferrer" target="_blank">https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/</a><br>
>>> [2]. <a href="http://aravindavk.in/blog/gluster-georep-tools/" rel="noreferrer" target="_blank">http://aravindavk.in/blog/gluster-georep-tools/</a><br>
>>><br>
>>><br>
>>> /Sunny<br>
>>><br>
>>> On Thu, Mar 21, 2019 at 11:28 AM Maurya M <<a href="mailto:mauryam@gmail.com" target="_blank">mauryam@gmail.com</a>> wrote:<br>
>>> ><br>
>>> > Hi Sunil,<br>
>>> > I did run the on the slave node :<br>
>>> > /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh azureuser vol_041afbc53746053368a1840607636e97 vol_a5aee81a873c043c99a938adcb5b5781<br>
>>> > getting this message "/home/azureuser/common_secret.pem.pub not present. Please run geo-replication command on master with push-pem option to generate the file"<br>
>>> ><br>
>>> > So went back and created the session again, no change, so manually copied the common_secret.pem.pub to /home/azureuser/ but still the set_geo_rep_pem_keys.sh is looking the pem file in different name : COMMON_SECRET_PEM_PUB=${master_vol}_${slave_vol}_common_secret.pem.pub , change the name of pem , ran the command again :<br>
>>> ><br>
>>> > /usr/libexec/glusterfs/set_geo_rep_pem_keys.sh azureuser vol_041afbc53746053368a1840607636e97 vol_a5aee81a873c043c99a938adcb5b5781<br>
>>> > Successfully copied file.<br>
>>> > Command executed successfully.<br>
>>> ><br>
>>> ><br>
>>> > - went back and created the session , start the geo-replication , still seeing the same error in logs. Any ideas ?<br>
>>> ><br>
>>> > thanks,<br>
>>> > Maurya<br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> > On Wed, Mar 20, 2019 at 11:07 PM Sunny Kumar <<a href="mailto:sunkumar@redhat.com" target="_blank">sunkumar@redhat.com</a>> wrote:<br>
>>> >><br>
>>> >> Hi Maurya,<br>
>>> >><br>
>>> >> I guess you missed last trick to distribute keys in slave node. I see<br>
>>> >> this is non-root geo-rep setup so please try this:<br>
>>> >><br>
>>> >><br>
>>> >> Run the following command as root in any one of Slave node.<br>
>>> >><br>
>>> >> /usr/local/libexec/glusterfs/set_geo_rep_pem_keys.sh <slave_user><br>
>>> >> <master_volume> <slave_volume><br>
>>> >><br>
>>> >> - Sunny<br>
>>> >><br>
>>> >> On Wed, Mar 20, 2019 at 10:47 PM Maurya M <<a href="mailto:mauryam@gmail.com" target="_blank">mauryam@gmail.com</a>> wrote:<br>
>>> >> ><br>
>>> >> > Hi all,<br>
>>> >> > Have setup a 3 master nodes - 3 slave nodes (gluster 4.1) for geo-replication, but once have the geo-replication configure the status is always on "Created',<br>
>>> >> > even after have force start the session.<br>
>>> >> ><br>
>>> >> > On close inspect of the logs on the master node seeing this error:<br>
>>> >> ><br>
>>> >> > "E [syncdutils(monitor):801:errlog] Popen: command returned error cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-replication/secret.pem -p 22 azureuser@xxxxx.xxxx..xxx. gluster --xml --remote-host=localhost volume info vol_a5ae34341a873c043c99a938adcb5b5781 error=255"<br>
>>> >> ><br>
>>> >> > Any ideas what is issue?<br>
>>> >> ><br>
>>> >> > thanks,<br>
>>> >> > Maurya<br>
>>> >> ><br>
>>> >> > _______________________________________________<br>
>>> >> > Gluster-users mailing list<br>
>>> >> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>>> >> > <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>
</blockquote></div>