<div dir="auto">Hi <div dir="auto">As discussed I have upgraded gluster from 4.1 to 6.2 version. But the Geo replication failed to start. </div><div dir="auto">Stays in faulty state</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019, 5:32 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Checked the data. It remains in 2708. No progress.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 4:36 PM Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank" rel="noreferrer">khiremat@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">That means it could be working and the defunct process might be some old zombie one. Could you check, that data progress ?<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 4:29 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" target="_blank" rel="noreferrer">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi <div>When i change the rsync option the rsync process doesnt seem to start . Only a defunt process is listed in ps aux. Only when i set rsync option to " " and restart all the process the rsync process is listed in ps aux.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 4:23 PM Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank" rel="noreferrer">khiremat@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Yes, rsync config option should have fixed this issue.</div><div><br></div><div>Could you share the output of the following?</div><div><br></div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">1. gluster volume geo-replication <MASTERVOL> <SLAVEHOST>::<SLAVEVOL> config rsync-options</span></div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">2. ps -ef | grep rsync </span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 4:11 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" target="_blank" rel="noreferrer">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Done. <br>We got the following result .<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">1559298781.338234 write(2, "rsync: link_stat \"/tmp/gsyncd-aux-mount-EEJ_sY/.gfid/3fa6aed8-802e-4efe-9903-8bc171176d88\" failed: No such file or directory (2)", 128</blockquote><div>seems like a file is missing ? </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 3:25 PM Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank" rel="noreferrer">khiremat@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>Could you take the strace with with more string size? The argument strings are truncated.</div><div><br></div><div>strace -s 500 -ttt -T -p <rsync pid> </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 3:17 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" target="_blank" rel="noreferrer">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Kotresh<div>The above-mentioned work around did not work properly.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 3:16 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" target="_blank" rel="noreferrer">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Kotresh <div>We have tried the above-mentioned rsync option and we are planning to have the version upgrade to 6.0.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 31, 2019 at 11:04 AM Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank" rel="noreferrer">khiremat@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>This looks like the hang because stderr buffer filled up with errors messages and no one reading it.</div><div>I think this issue is fixed in latest releases. As a workaround, you can do following and check if it works.</div><div><br></div><div>Prerequisite: <br></div><div> rsync version should be > 3.1.0</div><div><br></div><div>Workaround:</div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">gluster volume geo-replication <MASTERVOL> <SLAVEHOST>::<SLAVEVOL> config rsync-options "--<span class="m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail-m_5788142466170477461gmail-m_-409086624576408671gmail-m_-2058090265699260195gmail-m_-5501420080927683884gmail-m_-8680190711538730641gmail-il">ignore</span>-<span class="m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail-m_5788142466170477461gmail-m_-409086624576408671gmail-m_-2058090265699260195gmail-m_-5501420080927683884gmail-m_-8680190711538730641gmail-il">missing</span>-<span class="m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail-m_5788142466170477461gmail-m_-409086624576408671gmail-m_-2058090265699260195gmail-m_-5501420080927683884gmail-m_-8680190711538730641gmail-il">args</span>"</span></div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"><br></span></div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Thanks,<br></span></div><div><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:italic;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Kotresh HR<br></span></div><div><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 30, 2019 at 5:39 PM deepu srinivasan <<a href="mailto:sdeepugd@gmail.com" target="_blank" rel="noreferrer">sdeepugd@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="color:rgb(0,0,0);font-family:Verdana,Arial,Helvetica,sans-serif;font-size:13.3333px"><span style="color:rgb(34,34,34);font-family:Arial,Helvetica,sans-serif;font-size:small">Hi</span><br></div><div><div>We were evaluating Gluster geo Replication between two DCs one is in US west and one is in US east. We took multiple trials for different file size. <br></div><div>The Geo Replication tends to stop replicating but while checking the status it appears to be in Active state. But the slave volume did not increase in size.<br></div><div>So we have restarted the geo-replication session and checked the status. The status was in an active state and it was in History Crawl for a long time. We have enabled the DEBUG mode in logging and checked for any error.<br></div><div>There was around 2000 file appeared for syncing candidate. The Rsync process starts but the rsync did not happen in the slave volume. Every time the rsync process appears in the "ps auxxx" list but the replication did not happen in the slave end. What would be the cause of this problem? Is there anyway to debug it?<br></div><div><br></div><div>We have also checked the strace of the rync program.<br>it displays something like this </div><div><br>"<span style="background-color:transparent;color:rgb(21,21,21);font-family:inherit;font-size:15px;white-space:pre-wrap">write(2, "rsync: link_stat \"/tmp/gsyncd-au"..., 128</span>"</div><div><br></div><div><br></div><div>We are using the below specs<br></div><div><br></div><div>Gluster version - 4.1.7<br></div><div>Sync mode - rsync<br></div><div>Volume - 1x3 in each end (master and slave)<br></div><div>Intranet Bandwidth - 10 Gig</div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail-m_5788142466170477461gmail-m_-409086624576408671gmail-m_-2058090265699260195gmail-m_-5501420080927683884gmail-m_-8680190711538730641gmail-m_3584552368839152459gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail-m_5788142466170477461gmail-m_-409086624576408671gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="m_8387402863976225764gmail-m_-3853063166685410928gmail-m_-3968624705088776333gmail-m_-8857388651811668029gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="m_8387402863976225764gmail-m_-3853063166685410928gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</blockquote></div>
</blockquote></div>