<div dir="ltr"><div><div>It is clear that rsync is failing. Are the rsync versions on all masters and slave nodes same?<br></div>I have seen that has caused problems sometimes.<br><br></div>-Kotresh HR<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 24, 2018 at 10:29 PM, Dietmar Putz <span dir="ltr"><<a href="mailto:dietmar.putz@3qsdn.com" target="_blank">dietmar.putz@3qsdn.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hi all,</p>
i have made some tests on the latest Ubuntu 16.04.3 server image.
Upgrades were disabled...<br>
the configuration was always the same...a distributed replicated
volume on 4 VM's with geo-replication to a dist. repl .volume on 4
VM's.<br>
i started with 3.7.20, upgrade to 3.8.15, to 3.10.9 to 3.12.5. After
each upgrade i have tested the geo-replication which worked well
anytime.<br>
then i have made an update / upgrade on the first master node.
directly after upgrade the below shown error appeared on that node.<br>
after upgrade on the second master node the error appeared there
also... geo replication is faulty. <br>
<br>
this error affects gfs 3.7.20, 3.8.15, 3.10.9 and 3.12.5 on Ubuntu
16.04.3<br>
in one test i have updated rsync from 3.1.1 to 3.1.2 but with no
effect.<br>
<br>
does anyone else experienced this behavior...any idea ?<br>
<br>
best regards<br>
Dietmar<br>
<br>
<p><br>
</p>
<p><tt>gfs 3.12.5 geo-rep log on master :<br>
</tt></p>
<p><tt>[2018-01-24 15:50:35.347959] I
[master(/brick1/mvol1):1385:<wbr>crawl] _GMaster: slave's time
stime=(1516808792, 0)
</tt><tt><br>
</tt><tt>[2018-01-24 15:50:35.604094] I
[master(/brick1/mvol1):1863:<wbr>syncjob] Syncer: Sync Time Taken
duration=0.0294 num_files=1 job=2 return_code=3
</tt><tt><br>
</tt><tt>[2018-01-24 15:50:35.605490] E
[resource(/brick1/mvol1):210:<wbr>errlog] Popen: command returned
error cmd=rsync -aR0 --inplace --files-from=- --super --stats
--numeric-ids --no-implied-dirs --existing --xattrs --acls
--ignore-missing-args . -e ssh -oPasswordAuthentication=no
-oStrictHostKeyChecking=no -i
/var/lib/glusterd/geo-<wbr>replication/secret.pem -p 22
-oControlMaster=auto -S
/tmp/gsyncd-aux-ssh-MZwEp2/<wbr>cbad1c5f88978ecd713bdb1478fbab<wbr>be.sock
--compress root@gl-node5-int:/proc/2013/<wbr>cwd error=3
</tt><tt><br>
</tt><tt>[2018-01-24 15:50:35.628978] I
[syncdutils(/brick1/mvol1):<wbr>271:finalize] <top>: exiting.</tt></p>
<p><tt><br>
</tt></p>
<p><br>
</p>
<p>after this upgrade one server fails :<br>
<tt>Start-Date: 2018-01-18 04:33:52
<br>
Commandline: /usr/bin/unattended-upgrade
<br>
Upgrade:
<br>
libdns-export162:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
1:9.10.3.dfsg.P4-8ubuntu1.10),
<br>
libisccfg140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
1:9.10.3.dfsg.P4-8ubuntu1.10),
<br>
bind9-host:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
1:9.10.3.dfsg.P4-8ubuntu1.10),
<br>
dnsutils:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
1:9.10.3.dfsg.P4-8ubuntu1.10),
<br>
libc6:amd64 (2.23-0ubuntu9, 2.23-0ubuntu10),
<br>
libisc160:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
1:9.10.3.dfsg.P4-8ubuntu1.10),
<br>
locales:amd64 (2.23-0ubuntu9, 2.23-0ubuntu10),
<br>
libisc-export160:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
1:9.10.3.dfsg.P4-8ubuntu1.10),
<br>
libc-bin:amd64 (2.23-0ubuntu9, 2.23-0ubuntu10),
<br>
liblwres141:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
1:9.10.3.dfsg.P4-8ubuntu1.10),
<br>
libdns162:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
1:9.10.3.dfsg.P4-8ubuntu1.10),
<br>
multiarch-support:amd64 (2.23-0ubuntu9, 2.23-0ubuntu10),
<br>
libisccc140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
1:9.10.3.dfsg.P4-8ubuntu1.10),
<br>
libbind9-140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.8,
1:9.10.3.dfsg.P4-8ubuntu1.10)
<br>
End-Date: 2018-01-18 04:34:32
</tt></p>
<p><tt><br>
</tt></p>
<p><br>
<tt><tt>strace rsync :<br>
</tt></tt></p>
<p><tt><tt>30743 23:34:47 newfstatat(3, "6737",
{st_mode=S_IFDIR|0755, st_size=4096, ...},
AT_SYMLINK_NOFOLLOW) = 0<br>
30743 23:34:47 newfstatat(3, "6741", {st_mode=S_IFDIR|0755,
st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0<br>
30743 23:34:47 getdents(3, /* 0 entries */, 131072) = 0<br>
30743 23:34:47 munmap(0x7fa4feae7000, 135168) = 0<br>
30743 23:34:47 close(3) = 0<br>
30743 23:34:47 write(2, "rsync: getcwd(): No such file or
directory (2)", 46) = 46<br>
30743 23:34:47 write(2, "\n", 1) = 1<br>
30743 23:34:47 rt_sigaction(SIGUSR1, {SIG_IGN, [],
SA_RESTORER, 0x7fa4fdf404b0}, NULL, 8) = 0<br>
30743 23:34:47 rt_sigaction(SIGUSR2, {SIG_IGN, [],
SA_RESTORER, 0x7fa4fdf404b0}, NULL, 8) = 0<br>
30743 23:34:47 write(2, "rsync error: errors selecting
input/output files, dirs (code 3) at util.c(1056)
[Receiver=3.1.1]", 96) = 96<br>
30743 23:34:47 write(2, "\n", 1) = 1<br>
30743 23:34:47 exit_group(3) = ?<br>
30743 23:34:47 +++ exited with 3 +++</tt>
</tt></p>
<p><br>
</p>
<p><br>
</p>
<br>
<div class="m_3318466771008572220moz-cite-prefix">Am 19.01.2018 um 17:27 schrieb Joe
Julian:<br>
</div>
<blockquote type="cite">
<pre class="m_3318466771008572220k9mail">ubuntu 16.04</pre><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<pre class="m_3318466771008572220moz-signature" cols="72">--
Dietmar Putz
3Q GmbH
Kurfürstendamm 102
D-10711 Berlin
Mobile: +49 171 / 90 160 39
Mail: <a class="m_3318466771008572220moz-txt-link-abbreviated" href="mailto:dietmar.putz@3qsdn.com" target="_blank">dietmar.putz@3qsdn.com</a></pre>
</font></span></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</div>