<div dir="ltr"><div>Cool, just check whether they are hung by any chance with following command.<br><br></div>#strace -f -p 5921<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 2, 2018 at 12:25 PM, Marcus Pedersén <span dir="ltr"><<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="auto">On both active master nodes there is an rsync process. As in:
<div dir="auto"><br>
root 5921 0.0 0.0 115424 1176 ? S Aug01 0:00 rsync -aR0 --inplace --files-from=- --super --stats --numeric-ids --no-implied-dirs --xattrs --acls . -e ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-<wbr>replication/secret.pem
-p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-stuphs/<wbr>bf60c68f1a195dad59573a8dbaa309<wbr>f2.sock geouser@urd-gds-geo-001:/proc/<wbr>13077/cwd</div>
<div dir="auto"><br>
</div>
<div dir="auto">There is also ssh tunnels to slave nodes and gsyncd.py processes.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Regards</div><span class="">
<div dir="auto">Marcus </div>
<div dir="auto"><br>
<div dir="auto">################<br>
Marcus Pedersén<br>
Systemadministrator <br>
Interbull Centre<br>
################<br>
Sent from my phone <br>
################</div>
</div>
</span></div><div><div class="h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">Den 2 aug. 2018 08:07 skrev Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>:<br type="attribution">
</div>
</div>
<div>
<div dir="ltr">Could you look of any rsync processes hung in master or slave?<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, Aug 2, 2018 at 11:18 AM, Marcus Pedersén <span dir="ltr">
<<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="auto">
<div>Hi Kortesh,
<div dir="auto">rsync version 3.1.2 protocol version 31</div>
<div dir="auto">All nodes run CentOS 7, updated the last couple of days.</div>
<div dir="auto"><br>
</div>
<div dir="auto">Thanks</div>
<div dir="auto">Marcus <br>
<br>
<div dir="auto">################<span class="m_4366726778070998745HOEnZb"><font color="#888888"><br>
Marcus Pedersén<br>
Systemadministrator <br>
Interbull Centre<br>
################<br>
Sent from my phone <br>
################</font></span></div>
</div>
<div>
<div class="m_4366726778070998745h5"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">Den 2 aug. 2018 06:13 skrev Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>>:<br type="attribution">
<blockquote class="m_4366726778070998745m_9101179004808508027quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr">
<div>
<div>
<div>Hi Marcus,<br>
<br>
</div>
What's the rsync version being used?<br>
<br>
</div>
Thanks,<br>
</div>
Kotresh HR<br>
</div>
<div><br>
<div class="m_4366726778070998745m_9101179004808508027elided-text">On Thu, Aug 2, 2018 at 1:48 AM, Marcus Pedersén
<span dir="ltr"><<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>></span> wrote:<br>
<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:'calibri','arial','helvetica',sans-serif">
<p>Hi all!</p>
<p>I upgraded from 3.12.9 to 4.1.1 and had problems with geo-replication.</p>
<p>With help from the list with some sym links and so on (handled in another thread)</p>
<p>I got the geo-replication running.</p>
<p>It ran for 4-5 hours and then stopped, I stopped and started geo-replication and it ran for another 4-5 hours.</p>
<p>4.1.2 was released and I updated, hoping this would solve the problem.</p>
<p>I still have the same problem, at start it runs for 4-5 hours and then it stops.</p>
<p>After that nothing happens, I have waited for days but still nothing happens.</p>
<p><br>
</p>
<p>I have looked through logs but can not find anything obvious.<br>
</p>
<p><br>
</p>
<p>Status for geo-replication is <wbr>active for the two same nodes all the time:</p>
<p><br>
</p>
<p>MASTER NODE MASTER VOL MASTER BRICK SLAVE USER SLAVE <wbr> SLAVE NODE STATUS CRAWL STATUS LAST_SYNCED ENTRY DATA META FAILURES CHECKPOINT TIME CHECKPOINT
COMPLETED CHECKPOINT COMPLETION TIME <br>
------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>-----<br>
urd-gds-001 urd-gds-volume /urd-gds/gluster geouser geouser@urd-gds-geo-001::urd-g<wbr>ds-volume urd-gds-geo-000 Active History Crawl 2018-04-16 20:32:09 0 14205 0 0 2018-07-27 21:12:44 No
N/A <br>
urd-gds-002 urd-gds-volume /urd-gds/gluster geouser geouser@urd-gds-geo-001::urd-g<wbr>ds-volume urd-gds-geo-002 Passive N/A N/A N/A N/A N/A N/A N/A N/A
N/A <br>
urd-gds-004 urd-gds-volume /urd-gds/gluster geouser geouser@urd-gds-geo-001::urd-g<wbr>ds-volume urd-gds-geo-002 Passive N/A N/A N/A N/A N/A N/A N/A N/A
N/A <br>
urd-gds-003 urd-gds-volume /urd-gds/gluster geouser geouser@urd-gds-geo-001::urd-g<wbr>ds-volume urd-gds-geo-000 Active History Crawl 2018-05-01 20:58:14 285 4552 0 0 2018-07-27 21:12:44 No
N/A <br>
urd-gds-000 urd-gds-volume /urd-gds/gluster1 geouser geouser@urd-gds-geo-001::urd-g<wbr>ds-volume urd-gds-geo-001 Passive N/A N/A N/A N/A N/A N/A N/A N/A
N/A <br>
urd-gds-000 urd-gds-volume /urd-gds/gluster2 geouser geouser@urd-gds-geo-001::urd-g<wbr>ds-volume urd-gds-geo-001 Passive N/A N/A N/A N/A N/A N/A N/A N/A
N/A </p>
<p><br>
</p>
<p>Master cluster is Distribute-Replicate </p>
<p>2 x (2 + 1) </p>
<p>Used space 30TB<br>
</p>
<p><br>
</p>
<p>Slave cluster is Replicate</p>
<p>1 x (2 + 1)</p>
<p>Used space 9TB</p>
<p><br>
</p>
<p>Parts from gsyncd.logs are enclosed.<br>
</p>
<p><br>
</p>
<p>Thanks a lot!</p>
<p><br>
</p>
<p>Best regards</p>
<p>Marcus Pedersén</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<p>---<br>
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka
<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">
här </a><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click
<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">
here </a></p>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div>
<div dir="ltr">
<div>Thanks and Regards,<br>
</div>
Kotresh H R<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
<div>
<div class="m_4366726778070998745h5">
<p>---<br>
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka
<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">
här </a><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click
<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">
here </a></p>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="m_4366726778070998745gmail_signature">
<div dir="ltr">
<div>Thanks and Regards,<br>
</div>
Kotresh H R<br>
</div>
</div>
</div>
</div>
<p>---<br>
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka
<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">här </a><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click
<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">here </a></p>
</div></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</div>