<div dir="auto">Thankyou Kotresh</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 4, 2019, 11:20 PM Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com">khiremat@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Ccing Sunny, who was investing similar issue.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 4, 2019 at 5:46 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" target="_blank" rel="noreferrer">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Have already added the path in bashrc . Still in faulty state</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 4, 2019, 5:27 PM Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank" rel="noreferrer">khiremat@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">could you please try adding /usr/sbin to $PATH for user 'sas'? If it's bash, add 'export PATH=/usr/sbin:$PATH' in<div>/home/sas/.bashrc</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 4, 2019 at 5:24 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" rel="noreferrer noreferrer" target="_blank">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Kortesh <br></div><div>Please find the logs of the above error<u><b><br></b></u></div><div><u><b>Master log snippet</b></u></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>[2019-06-04 11:52:09.254731] I [resource(worker /home/sas/gluster/data/code-misc):1379:connect_remote] SSH: Initializing SSH connection between master and slave...<br> [2019-06-04 11:52:09.308923] D [repce(worker /home/sas/gluster/data/code-misc):196:push] RepceClient: call 89724:139652759443264:1559649129.31 __repce_version__() ...<br> [2019-06-04 11:52:09.602792] E [syncdutils(worker /home/sas/gluster/data/code-misc):311:log_raise_exception] <top>: connection to peer is broken<br> [2019-06-04 11:52:09.603312] E [syncdutils(worker /home/sas/gluster/data/code-misc):805:errlog] Popen: command returned error cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/ glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-4aL2tc/d893f66e0addc32f7d0080bb503f5185.sock <a href="mailto:sas@192.168.185.107" rel="noreferrer noreferrer" target="_blank">sas@192.168.185.107</a> /usr/libexec/glusterfs/gsyncd slave code-misc sas@ 192.168.185.107::code-misc --master-node 192.168.185.106 --master-node-id 851b64d0-d885-4ae9-9b38-ab5b15db0fec --master-brick /home/sas/gluster/data/code-misc --local-node 192.168.185.122 --local-node- id bcaa7af6-c3a1-4411-8e99-4ebecb32eb6a --slave-timeout 120 --slave-log-level DEBUG --slave-gluster-log-level INFO --slave-gluster-command-dir /usr/sbin error=1<br> [2019-06-04 11:52:09.614996] I [repce(agent /home/sas/gluster/data/code-misc):97:service_loop] RepceServer: terminating on reaching EOF.<br> [2019-06-04 11:52:09.615545] D [monitor(monitor):271:monitor] Monitor: worker(/home/sas/gluster/data/code-misc) connected<br> [2019-06-04 11:52:09.616528] I [monitor(monitor):278:monitor] Monitor: worker died in startup phase brick=/home/sas/gluster/data/code-misc<br> [2019-06-04 11:52:09.619391] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty<br></div></blockquote><div><br></div><div><u><b>Slave log snippet</b></u><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[2019-06-04 11:50:09.782668] E [syncdutils(slave
<a href="http://192.168.185.106/home/sas/gluster/data/code-misc):809:logerr" rel="noreferrer noreferrer" target="_blank">192.168.185.106/home/sas/gluster/data/code-misc):809:logerr</a>] Popen:
/usr/sbin/gluster> 2 : failed with this errno (No such file or
directory)<br>[2019-06-04 11:50:11.188167] W [gsyncd(slave
<a href="http://192.168.185.125/home/sas/gluster/data/code-misc):305:main" rel="noreferrer noreferrer" target="_blank">192.168.185.125/home/sas/gluster/data/code-misc):305:main</a>] <top>:
Session config file not exists, using the default config
path=/var/lib/glusterd/geo-replication/code-misc_192.168.185.107_code-misc/gsyncd.conf<br>[2019-06-04
11:50:11.201070] I [resource(slave
<a href="http://192.168.185.125/home/sas/gluster/data/code-misc):1098:connect" rel="noreferrer noreferrer" target="_blank">192.168.185.125/home/sas/gluster/data/code-misc):1098:connect</a>] GLUSTER:
Mounting gluster volume locally...<br>[2019-06-04 11:50:11.271231] E
[resource(slave
<a href="http://192.168.185.125/home/sas/gluster/data/code-misc):1006:handle_mounter" rel="noreferrer noreferrer" target="_blank">192.168.185.125/home/sas/gluster/data/code-misc):1006:handle_mounter</a>]
MountbrokerMounter: glusterd answered mnt=<br>[2019-06-04
11:50:11.271998] E [syncdutils(slave
<a href="http://192.168.185.125/home/sas/gluster/data/code-misc):805:errlog" rel="noreferrer noreferrer" target="_blank">192.168.185.125/home/sas/gluster/data/code-misc):805:errlog</a>] Popen:
command returned error cmd=/usr/sbin/gluster --remote-host=localhost
system:: mount sas user-map-root=sas aux-gfid-mount acl log-level=INFO
log-file=/var/log/glusterfs/geo-replication-slaves/code-misc_192.168.185.107_code-misc/mnt-192.168.185.125-home-sas-gluster-data-code-misc.log
volfile-server=localhost volfile-id=code-misc client-pid=-1 error=1<br>[2019-06-04
11:50:11.272113] E [syncdutils(slave
<a href="http://192.168.185.125/home/sas/gluster/data/code-misc):809:logerr" rel="noreferrer noreferrer" target="_blank">192.168.185.125/home/sas/gluster/data/code-misc):809:logerr</a>] Popen:
/usr/sbin/gluster> 2 : failed with this errno (No such file or
directory)</blockquote></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 4, 2019 at 5:10 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" rel="noreferrer noreferrer" target="_blank">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Hi <div dir="auto">As discussed I have upgraded gluster from 4.1 to 6.2 version. But the Geo replication failed to start. </div><div dir="auto">Stays in faulty state</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019, 5:32 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" rel="noreferrer noreferrer" target="_blank">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Checked the data. It remains in 2708. No progress.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" rel="noreferrer noreferrer noreferrer" target="_blank">khiremat@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">That means it could be working and the defunct process might be some old zombie one. Could you check, that data progress ?<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 4:29 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" rel="noreferrer noreferrer noreferrer" target="_blank">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi <div>When i change the rsync option the rsync process doesnt seem to start . Only a defunt process is listed in ps aux. Only when i set rsync option to " " and restart all the process the rsync process is listed in ps aux.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" rel="noreferrer noreferrer noreferrer" target="_blank">khiremat@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Yes, rsync config option should have fixed this issue.</div><div><br></div><div>Could you share the output of the following?</div><div><br></div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">1. gluster volume geo-replication <MASTERVOL> <SLAVEHOST>::<SLAVEVOL> config rsync-options</span></div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">2. ps -ef | grep rsync </span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 4:11 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" rel="noreferrer noreferrer noreferrer" target="_blank">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Done. <br>We got the following result .<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">1559298781.338234 write(2, "rsync: link_stat \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\" failed: No such file or directory (2)", 128</blockquote><div>seems like a file is missing ? </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 3:25 PM Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" rel="noreferrer noreferrer noreferrer" target="_blank">khiremat@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>Could you take the strace with with more string size? The argument strings are truncated.</div><div><br></div><div>strace -s 500 -ttt -T -p <rsync pid> </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 3:17 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" rel="noreferrer noreferrer noreferrer" target="_blank">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Kotresh<div>The above-mentioned work around did not work properly.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 3:16 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" rel="noreferrer noreferrer noreferrer" target="_blank">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Kotresh <div>We have tried the above-mentioned rsync option and we are planning to have the version upgrade to 6.0.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" rel="noreferrer noreferrer noreferrer" target="_blank">khiremat@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>This looks like the hang because stderr buffer filled up with errors messages and no one reading it.</div><div>I think this issue is fixed in latest releases. As a workaround, you can do following and check if it works.</div><div><br></div><div>Prerequisite: <br></div><div> rsync version should be > 3.1.0</div><div><br></div><div>Workaround:</div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">gluster volume geo-replication <MASTERVOL> <SLAVEHOST>::<SLAVEVOL> config rsync-options "--<span class="m_2084022293248688888gmail-m_-2798290358597502776m_-4115114979199400579gmail-m_-3490639267642599601gmail-m_-8861568382436114415m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail-m_5788142466170477461gmail-m_-409086624576408671gmail-m_-2058090265699260195gmail-m_-5501420080927683884gmail-m_-8680190711538730641gmail-il">ignore</span>-<span class="m_2084022293248688888gmail-m_-2798290358597502776m_-4115114979199400579gmail-m_-3490639267642599601gmail-m_-8861568382436114415m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail-m_5788142466170477461gmail-m_-409086624576408671gmail-m_-2058090265699260195gmail-m_-5501420080927683884gmail-m_-8680190711538730641gmail-il">missing</span>-<span class="m_2084022293248688888gmail-m_-2798290358597502776m_-4115114979199400579gmail-m_-3490639267642599601gmail-m_-8861568382436114415m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail-m_5788142466170477461gmail-m_-409086624576408671gmail-m_-2058090265699260195gmail-m_-5501420080927683884gmail-m_-8680190711538730641gmail-il">args</span>"</span></div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><br></span></div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Thanks,<br></span></div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Kotresh HR<br></span></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 30, 2019 at 5:39 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" rel="noreferrer noreferrer noreferrer" target="_blank">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:13.3333px"><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;font-size:small">Hi</span><br></div><div><div>We were evaluating Gluster geo Replication between two DCs one is in US west and one is in US east. We took multiple trials for different file size. <br></div><div>The Geo Replication tends to stop replicating but while checking the status it appears to be in Active state. But the slave volume did not increase in size.<br></div><div>So we have restarted the geo-replication session and checked the status. The status was in an active state and it was in History Crawl for a long time. We have enabled the DEBUG mode in logging and checked for any error.<br></div><div>There was around 2000 file appeared for syncing candidate. The Rsync process starts but the rsync did not happen in the slave volume. Every time the rsync process appears in the "ps auxxx" list but the replication did not happen in the slave end. What would be the cause of this problem? Is there anyway to debug it?<br></div><div><br></div><div>We have also checked the strace of the rync program.<br>it displays something like this </div><div><br>"<span style="background-color:transparent;color:rgb(21,21,21);font-family:inherit;font-size:15px;white-space:pre-wrap">write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128</span>"</div><div><br></div><div><br></div><div>We are using the below specs<br></div><div><br></div><div>Gluster version - 4.1.7<br></div><div>Sync mode - rsync<br></div><div>Volume - 1x3 in each end (master and slave)<br></div><div>Intranet Bandwidth - 10 Gig</div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="m_2084022293248688888gmail-m_-2798290358597502776m_-4115114979199400579gmail-m_-3490639267642599601gmail-m_-8861568382436114415m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail-m_5788142466170477461gmail-m_-409086624576408671gmail-m_-2058090265699260195gmail-m_-5501420080927683884gmail-m_-8680190711538730641gmail-m_3584552368839152459gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="m_2084022293248688888gmail-m_-2798290358597502776m_-4115114979199400579gmail-m_-3490639267642599601gmail-m_-8861568382436114415m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail-m_5788142466170477461gmail-m_-409086624576408671gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="m_2084022293248688888gmail-m_-2798290358597502776m_-4115114979199400579gmail-m_-3490639267642599601gmail-m_-8861568382436114415m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="m_2084022293248688888gmail-m_-2798290358597502776m_-4115114979199400579gmail-m_-3490639267642599601gmail-m_-8861568382436114415m_8387402863976225764gmail-m_-3853063166685410928gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_2084022293248688888gmail-m_-2798290358597502776m_-4115114979199400579gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_2084022293248688888gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</blockquote></div>