<div dir="ltr"><div dir="ltr">some addtion logs from gverify-mastermnt.log &amp; gverify-slavemnt.log: <div><br></div><div><div>[2019-03-25 12:13:23.819665] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-vol_75a5fd373d88ba687f591f3353fa05cf-client-2: error returned while attempting to connect to host:(null), port:0</div><div>[2019-03-25 12:13:23.819814] W [dict.c:923:str_to_data] (--&gt;/usr/lib64/glusterfs/4.1.7/xlator/protocol/client.so(+0x40c0a) [0x7f3eb4d86c0a] --&gt;/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f3ebc334266] --&gt;/lib64/libglusterfs.so.0(str_to_data+0x91) [0x7f3ebc330ea1] ) 0-dict: <b>value is NULL [Invalid argument]</b></div></div><div><b><br></b></div><div><b><br></b></div><div> any idea how to fix this ? any patch file i can try with please share.</div><div><br></div><div>thanks,</div><div>Maurya</div><div><b><br></b></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 25, 2019 at 3:37 PM Maurya M &lt;<a href="mailto:mauryam@gmail.com">mauryam@gmail.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">ran this command :  ssh -p 2222 -i /var/lib/glusterd/geo-replication/secret.pem root@&lt;slave node&gt;gluster volume info --xml </div><div dir="ltr"><br></div><div dir="ltr">attaching the output.<br><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Mar 25, 2019 at 2:13 PM Aravinda &lt;<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Geo-rep is running `ssh -i /var/lib/glusterd/geo-replication/secret.pem <br>
root@&lt;slavenode&gt; gluster volume info --xml` and parsing its output.<br>
Please try to to run the command from the same node and let us know the<br>
output.<br>
<br>
<br>
On Mon, 2019-03-25 at 11:43 +0530, Maurya M wrote:<br>
&gt; Now the error is on the same line 860 : as highlighted below:<br>
&gt; <br>
&gt; [2019-03-25 06:11:52.376238] E<br>
&gt; [syncdutils(monitor):332:log_raise_exception] &lt;top&gt;: FAIL:<br>
&gt; Traceback (most recent call last):<br>
&gt;   File &quot;/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py&quot;, line<br>
&gt; 311, in main<br>
&gt;     func(args)<br>
&gt;   File &quot;/usr/libexec/glusterfs/python/syncdaemon/subcmds.py&quot;, line<br>
&gt; 50, in subcmd_monitor<br>
&gt;     return monitor.monitor(local, remote)<br>
&gt;   File &quot;/usr/libexec/glusterfs/python/syncdaemon/monitor.py&quot;, line<br>
&gt; 427, in monitor<br>
&gt;     return Monitor().multiplex(*distribute(local, remote))<br>
&gt;   File &quot;/usr/libexec/glusterfs/python/syncdaemon/monitor.py&quot;, line<br>
&gt; 386, in distribute<br>
&gt;     svol = Volinfo(slave.volume, &quot;localhost&quot;, prelude)<br>
&gt;   File &quot;/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py&quot;, line<br>
&gt; 860, in __init__<br>
&gt;     vi = XET.fromstring(vix)<br>
&gt;   File &quot;/usr/lib64/python2.7/xml/etree/ElementTree.py&quot;, line 1300, in<br>
&gt; XML<br>
&gt;     parser.feed(text)<br>
&gt;   File &quot;/usr/lib64/python2.7/xml/etree/ElementTree.py&quot;, line 1642, in<br>
&gt; feed<br>
&gt;     self._raiseerror(v)<br>
&gt;   File &quot;/usr/lib64/python2.7/xml/etree/ElementTree.py&quot;, line 1506, in<br>
&gt; _raiseerror<br>
&gt;     raise err<br>
&gt; ParseError: syntax error: line 1, column 0<br>
&gt; <br>
&gt; <br>
&gt; On Mon, Mar 25, 2019 at 11:29 AM Maurya M &lt;<a href="mailto:mauryam@gmail.com" target="_blank">mauryam@gmail.com</a>&gt; wrote:<br>
&gt; &gt; Sorry my bad, had put the print line to debug, i am using gluster<br>
&gt; &gt; 4.1.7, will remove the print line.<br>
&gt; &gt; <br>
&gt; &gt; On Mon, Mar 25, 2019 at 10:52 AM Aravinda &lt;<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>&gt;<br>
&gt; &gt; wrote:<br>
&gt; &gt; &gt; Below print statement looks wrong. Latest Glusterfs code doesn&#39;t<br>
&gt; &gt; &gt; have<br>
&gt; &gt; &gt; this print statement. Please let us know which version of<br>
&gt; &gt; &gt; glusterfs you<br>
&gt; &gt; &gt; are using.<br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt; ```<br>
&gt; &gt; &gt;   File &quot;/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py&quot;,<br>
&gt; &gt; &gt; line<br>
&gt; &gt; &gt; 860, in __init__<br>
&gt; &gt; &gt;     print &quot;debug varible &quot; %vix<br>
&gt; &gt; &gt; ```<br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt; As a workaround, edit that file and comment the print line and<br>
&gt; &gt; &gt; test the<br>
&gt; &gt; &gt; geo-rep config command.<br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt; <br>
&gt; &gt; &gt; On Mon, 2019-03-25 at 09:46 +0530, Maurya M wrote:<br>
&gt; &gt; &gt; &gt; hi Aravinda,<br>
&gt; &gt; &gt; &gt;  had the session created using : create ssh-port 2222 push-pem<br>
&gt; &gt; &gt; and<br>
&gt; &gt; &gt; &gt; also the :<br>
&gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; gluster volume geo-replication<br>
&gt; &gt; &gt; vol_75a5fd373d88ba687f591f3353fa05cf<br>
&gt; &gt; &gt; &gt; 172.16.201.35::vol_e783a730578e45ed9d51b9a80df6c33f config ssh-<br>
&gt; &gt; &gt; port<br>
&gt; &gt; &gt; &gt; 2222<br>
&gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; hitting this message:<br>
&gt; &gt; &gt; &gt; geo-replication config-set failed for<br>
&gt; &gt; &gt; &gt; vol_75a5fd373d88ba687f591f3353fa05cf<br>
&gt; &gt; &gt; &gt; 172.16.201.35::vol_e783a730578e45ed9d51b9a80df6c33f<br>
&gt; &gt; &gt; &gt; geo-replication command failed<br>
&gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; Below is snap of status:<br>
&gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; [root@k8s-agentpool1-24779565-1<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vol_e783a73057<br>
&gt; &gt; &gt; 8e45ed9d51b9a80df6c33f]# gluster volume geo-replication<br>
&gt; &gt; &gt; vol_75a5fd373d88ba687f591f3353fa05cf<br>
&gt; &gt; &gt; 172.16.201.35::vol_e783a730578e45ed9d51b9a80df6c33f status<br>
&gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; MASTER NODE      MASTER VOL                              MASTER<br>
&gt; &gt; &gt; &gt; BRICK                                                         <br>
&gt; &gt; &gt;      <br>
&gt; &gt; &gt; &gt;                                SLAVE USER    SLAVE             <br>
&gt; &gt; &gt;      <br>
&gt; &gt; &gt; &gt;                               SLAVE NODE    STATUS     CRAWL<br>
&gt; &gt; &gt; STATUS <br>
&gt; &gt; &gt; &gt;   LAST_SYNCED<br>
&gt; &gt; &gt; &gt; -------------------------------------------------------------<br>
&gt; &gt; &gt; ------<br>
&gt; &gt; &gt; &gt; -------------------------------------------------------------<br>
&gt; &gt; &gt; ------<br>
&gt; &gt; &gt; &gt; -------------------------------------------------------------<br>
&gt; &gt; &gt; ------<br>
&gt; &gt; &gt; &gt; -------------------------------------------------------------<br>
&gt; &gt; &gt; ------<br>
&gt; &gt; &gt; &gt; ----------------<br>
&gt; &gt; &gt; &gt; 172.16.189.4     vol_75a5fd373d88ba687f591f3353fa05cf   <br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; /var/lib/heketi/mounts/vg_aee3df7b0bb2451bc00a73358c5196a2/brick_<br>
&gt; &gt; &gt; 116f<br>
&gt; &gt; &gt; &gt; b9427fb26f752d9ba8e45e183cb1/brick    root         <br>
&gt; &gt; &gt; &gt; 172.16.201.35::vol_e783a730578e45ed9d51b9a80df6c33f    N/A     <br>
&gt; &gt; &gt;    <br>
&gt; &gt; &gt; &gt;  Created    N/A             N/A<br>
&gt; &gt; &gt; &gt; 172.16.189.35    vol_75a5fd373d88ba687f591f3353fa05cf   <br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; /var/lib/heketi/mounts/vg_05708751110fe60b3e7da15bdcf6d4d4/brick_<br>
&gt; &gt; &gt; 266b<br>
&gt; &gt; &gt; &gt; b08f0d466d346f8c0b19569736fb/brick    root         <br>
&gt; &gt; &gt; &gt; 172.16.201.35::vol_e783a730578e45ed9d51b9a80df6c33f    N/A     <br>
&gt; &gt; &gt;    <br>
&gt; &gt; &gt; &gt;  Created    N/A             N/A<br>
&gt; &gt; &gt; &gt; 172.16.189.66    vol_75a5fd373d88ba687f591f3353fa05cf   <br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; /var/lib/heketi/mounts/vg_4b92a2b687e59b7311055d3809b77c06/brick_<br>
&gt; &gt; &gt; dfa4<br>
&gt; &gt; &gt; &gt; 4c9380cdedac708e27e2c2a443a0/brick    root         <br>
&gt; &gt; &gt; &gt; 172.16.201.35::vol_e783a730578e45ed9d51b9a80df6c33f    N/A     <br>
&gt; &gt; &gt;    <br>
&gt; &gt; &gt; &gt;  Created    N/A             N/A<br>
&gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; any ideas ? where can find logs for the failed commands check<br>
&gt; &gt; &gt; in<br>
&gt; &gt; &gt; &gt; gysncd.log , the trace is as below:<br>
&gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; [2019-03-25 04:04:42.295043] I [gsyncd(monitor):297:main]<br>
&gt; &gt; &gt; &lt;top&gt;:<br>
&gt; &gt; &gt; &gt; Using session config file      path=/var/lib/glusterd/geo-<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; replication/vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vo<br>
&gt; &gt; &gt; l_e7<br>
&gt; &gt; &gt; &gt; 83a730578e45ed9d51b9a80df6c33f/gsyncd.conf<br>
&gt; &gt; &gt; &gt; [2019-03-25 04:04:42.387192] E<br>
&gt; &gt; &gt; &gt; [syncdutils(monitor):332:log_raise_exception] &lt;top&gt;: FAIL:<br>
&gt; &gt; &gt; &gt; Traceback (most recent call last):<br>
&gt; &gt; &gt; &gt;   File &quot;/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py&quot;,<br>
&gt; &gt; &gt; line<br>
&gt; &gt; &gt; &gt; 311, in main<br>
&gt; &gt; &gt; &gt;     func(args)<br>
&gt; &gt; &gt; &gt;   File &quot;/usr/libexec/glusterfs/python/syncdaemon/subcmds.py&quot;,<br>
&gt; &gt; &gt; line<br>
&gt; &gt; &gt; &gt; 50, in subcmd_monitor<br>
&gt; &gt; &gt; &gt;     return monitor.monitor(local, remote)<br>
&gt; &gt; &gt; &gt;   File &quot;/usr/libexec/glusterfs/python/syncdaemon/monitor.py&quot;,<br>
&gt; &gt; &gt; line<br>
&gt; &gt; &gt; &gt; 427, in monitor<br>
&gt; &gt; &gt; &gt;     return Monitor().multiplex(*distribute(local, remote))<br>
&gt; &gt; &gt; &gt;   File &quot;/usr/libexec/glusterfs/python/syncdaemon/monitor.py&quot;,<br>
&gt; &gt; &gt; line<br>
&gt; &gt; &gt; &gt; 370, in distribute<br>
&gt; &gt; &gt; &gt;     mvol = Volinfo(master.volume, master.host)<br>
&gt; &gt; &gt; &gt;   File<br>
&gt; &gt; &gt; &quot;/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py&quot;, line<br>
&gt; &gt; &gt; &gt; 860, in __init__<br>
&gt; &gt; &gt; &gt;     print &quot;debug varible &quot; %vix<br>
&gt; &gt; &gt; &gt; TypeError: not all arguments converted during string formatting<br>
&gt; &gt; &gt; &gt; [2019-03-25 04:04:48.997519] I [gsyncd(config-get):297:main]<br>
&gt; &gt; &gt; &lt;top&gt;:<br>
&gt; &gt; &gt; &gt; Using session config file   path=/var/lib/glusterd/geo-<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; replication/vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vo<br>
&gt; &gt; &gt; l_e7<br>
&gt; &gt; &gt; &gt; 83a730578e45ed9d51b9a80df6c33f/gsyncd.conf<br>
&gt; &gt; &gt; &gt; [2019-03-25 04:04:49.93528] I [gsyncd(status):297:main] &lt;top&gt;:<br>
&gt; &gt; &gt; Using<br>
&gt; &gt; &gt; &gt; session config file        path=/var/lib/glusterd/geo-<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; replication/vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vo<br>
&gt; &gt; &gt; l_e7<br>
&gt; &gt; &gt; &gt; 83a730578e45ed9d51b9a80df6c33f/gsyncd.conf<br>
&gt; &gt; &gt; &gt; [2019-03-25 04:08:07.194348] I [gsyncd(config-get):297:main]<br>
&gt; &gt; &gt; &lt;top&gt;:<br>
&gt; &gt; &gt; &gt; Using session config file   path=/var/lib/glusterd/geo-<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; replication/vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vo<br>
&gt; &gt; &gt; l_e7<br>
&gt; &gt; &gt; &gt; 83a730578e45ed9d51b9a80df6c33f/gsyncd.conf<br>
&gt; &gt; &gt; &gt; [2019-03-25 04:08:07.262588] I [gsyncd(config-get):297:main]<br>
&gt; &gt; &gt; &lt;top&gt;:<br>
&gt; &gt; &gt; &gt; Using session config file   path=/var/lib/glusterd/geo-<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; replication/vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vo<br>
&gt; &gt; &gt; l_e7<br>
&gt; &gt; &gt; &gt; 83a730578e45ed9d51b9a80df6c33f/gsyncd.conf<br>
&gt; &gt; &gt; &gt; [2019-03-25 04:08:07.550080] I [gsyncd(config-get):297:main]<br>
&gt; &gt; &gt; &lt;top&gt;:<br>
&gt; &gt; &gt; &gt; Using session config file   path=/var/lib/glusterd/geo-<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; replication/vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vo<br>
&gt; &gt; &gt; l_e7<br>
&gt; &gt; &gt; &gt; 83a730578e45ed9d51b9a80df6c33f/gsyncd.conf<br>
&gt; &gt; &gt; &gt; [2019-03-25 04:08:18.933028] I [gsyncd(config-get):297:main]<br>
&gt; &gt; &gt; &lt;top&gt;:<br>
&gt; &gt; &gt; &gt; Using session config file   path=/var/lib/glusterd/geo-<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; replication/vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vo<br>
&gt; &gt; &gt; l_e7<br>
&gt; &gt; &gt; &gt; 83a730578e45ed9d51b9a80df6c33f/gsyncd.conf<br>
&gt; &gt; &gt; &gt; [2019-03-25 04:08:19.25285] I [gsyncd(status):297:main] &lt;top&gt;:<br>
&gt; &gt; &gt; Using<br>
&gt; &gt; &gt; &gt; session config file        path=/var/lib/glusterd/geo-<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; replication/vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vo<br>
&gt; &gt; &gt; l_e7<br>
&gt; &gt; &gt; &gt; 83a730578e45ed9d51b9a80df6c33f/gsyncd.conf<br>
&gt; &gt; &gt; &gt; [2019-03-25 04:09:15.766882] I [gsyncd(config-get):297:main]<br>
&gt; &gt; &gt; &lt;top&gt;:<br>
&gt; &gt; &gt; &gt; Using session config file   path=/var/lib/glusterd/geo-<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; replication/vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vo<br>
&gt; &gt; &gt; l_e7<br>
&gt; &gt; &gt; &gt; 83a730578e45ed9d51b9a80df6c33f/gsyncd.conf<br>
&gt; &gt; &gt; &gt; [2019-03-25 04:09:16.30267] I [gsyncd(config-get):297:main]<br>
&gt; &gt; &gt; &lt;top&gt;:<br>
&gt; &gt; &gt; &gt; Using session config file    path=/var/lib/glusterd/geo-<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; replication/vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vo<br>
&gt; &gt; &gt; l_e7<br>
&gt; &gt; &gt; &gt; 83a730578e45ed9d51b9a80df6c33f/gsyncd.conf<br>
&gt; &gt; &gt; &gt; [2019-03-25 04:09:16.89006] I [gsyncd(config-set):297:main]<br>
&gt; &gt; &gt; &lt;top&gt;:<br>
&gt; &gt; &gt; &gt; Using session config file    path=/var/lib/glusterd/geo-<br>
&gt; &gt; &gt; &gt;<br>
&gt; &gt; &gt; replication/vol_75a5fd373d88ba687f591f3353fa05cf_172.16.201.35_vo<br>
&gt; &gt; &gt; l_e7<br>
&gt; &gt; &gt; &gt; 83a730578e45ed9d51b9a80df6c33f/gsyncd.conf<br>
&gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; regards,<br>
&gt; &gt; &gt; &gt; Maurya<br>
&gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; On Mon, Mar 25, 2019 at 9:08 AM Aravinda &lt;<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>&gt;<br>
&gt; &gt; &gt; wrote:<br>
&gt; &gt; &gt; &gt; &gt; Use `ssh-port &lt;port&gt;` while creating the Geo-rep session<br>
&gt; &gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; &gt; Ref: <br>
&gt; &gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; <a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/#creating-the-session" rel="noreferrer" target="_blank">https://docs.gluster.org/en/latest/Administrator%20Guide/Geo%20Replication/#creating-the-session</a><br>
&gt; &gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; &gt; And set the ssh-port option before start.<br>
&gt; &gt; &gt; &gt; &gt; <br>
&gt; &gt; &gt; &gt; &gt; ```<br>
&gt; &gt; &gt; &gt; &gt; gluster volume geo-replication &lt;master_volume&gt; \<br>
&gt; &gt; &gt; &gt; &gt;     [&lt;slave_user&gt;@]&lt;slave_host&gt;::&lt;slave_volume&gt; config<br>
&gt; &gt; &gt; &gt; &gt;     ssh-port 2222<br>
&gt; &gt; &gt; &gt; &gt; ```<br>
&gt; &gt; &gt; &gt; &gt; <br>
-- <br>
regards<br>
Aravinda<br>
<br>
</blockquote></div>
</blockquote></div>