<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 2, 2021 at 8:14 PM Michael Scherer <<a href="mailto:mscherer@redhat.com">mscherer@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
so we finally found the cause of the georep failure, after several days<br>
of work from Deepshika and I. <br>
<br>
Short story:<br>
============<br>
<br>
side effect of adding libtirpc-devel on EL 7:<br>
<a href="https://github.com/gluster/project-infrastructure/issues/115" rel="noreferrer" target="_blank">https://github.com/gluster/project-infrastructure/issues/115</a></blockquote><div><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Looking at <a href="https://github.com/gluster/glusterfs-patch-acceptance-tests/pull/191">https://github.com/gluster/glusterfs-patch-acceptance-tests/pull/191</a> - we weren't supposed to use it?</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">From <a href="https://github.com/gluster/glusterfs/blob/d1d7a6f35c816822fab51c820e25023863c239c1/glusterfs.spec.in#L61">https://github.com/gluster/glusterfs/blob/d1d7a6f35c816822fab51c820e25023863c239c1/glusterfs.spec.in#L61</a> :</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"># Do not use libtirpc on EL6, it does not have xdr_uint64_t() and xdr_uint32_t<br># Do not use libtirpc on EL7, it does not have xdr_sizeof()<br>%if ( 0%{?rhel} && 0%{?rhel} <= 7 )<br>%global _without_libtirpc --without-libtirpc<br>%endif<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">CentOS 7 has an ancient version, CentOS 8 has a newer version, so perhaps just one CentOS 8 slaves?</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Y.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
Long story:<br>
===========<br>
<br>
So we first puzzled on why it was failing just on some builders and not<br>
others, especially since it was working fine on softserve VMs. <br>
<br>
We tried to look for the usual suspect, rebooted, reinstalled, searched<br>
if there was something weird (too much ssh keys, not enough inode, some<br>
hardware issue), but nothing obvious. <br>
<br>
After trying to find my way in the logs file and a few weird leads<br>
(like, why gsyncd was running gcc ? (answer: ctypes)), I was left with<br>
a rather cryptic message:<br>
<br>
[2021-02-02 15:19:00.040817 +0000] I<br>
[socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing<br>
(AF_UNIX) reuse check socket 18<br>
[2021-02-02 15:19:02.041641 +0000] W [xdr-<br>
rpcclnt.c:68:rpc_request_to_xdr] 0-rpc: failed to encode call msg<br>
[2021-02-02 15:19:02.041673 +0000] E [rpc-<br>
clnt.c:1507:rpc_clnt_record_build_record] 0-gfchangelog: Failed to<br>
build record header<br>
[2021-02-02 15:19:02.041683 +0000] W [rpc-clnt.c:1664:rpc_clnt_submit]<br>
0-gfchangelog: cannot build rpc-record<br>
[2021-02-02 15:19:02.041692 +0000] E [MSGID: 132023] [gf-<br>
changelog.c:285:gf_changelog_setup_rpc] 0-gfchangelog: Could not<br>
initiate probe RPC, bailing out!!! <br>
[2021-02-02 15:19:02.041809 +0000] E [MSGID: 132022] [gf-<br>
changelog.c:583:gf_changelog_register_generic] 0-gfchangelog: Error<br>
registering with changelog xlator <br>
<br>
Given that all gluster is around RPC, it would be unlikely that rpc is<br>
broken, but that's the only messages we had.<br>
<br>
<br>
We also found that the only builder that was working was builder 210.<br>
Upon looking, we found that 210 failed to be updated with ansible, due<br>
to some debugging we forgot to revert, which made this task fail: <br>
<a href="https://github.com/gluster/gluster.org_ansible_configuration/blob/master/roles/gluster_qa_scripts/tasks/main.yml#L7" rel="noreferrer" target="_blank">https://github.com/gluster/gluster.org_ansible_configuration/blob/master/roles/gluster_qa_scripts/tasks/main.yml#L7</a><br>
<br>
But it wasn't clear how that would change anything, since the only diff<br>
was a "set -e" that wasn't removed.<br>
<br>
Then Deepshika started to test more than georep, and she noticed that a<br>
lot of others tests were failing, with the same exact message about<br>
rpc. <br>
<br>
And she started to wonder if anything was recently changed. And indeed:<br>
<br>
# rpm -qa --last | head -n 15<br>
yum-plugin-auto-update-debug-info-1.1.31-54.el7_8.noarch mar. 02 févr.<br>
2021 14:04:59 UTC<br>
python3-debuginfo-3.6.8-18.el7.x86_64 mar. 02 févr. 2021<br>
14:04:58 UTC<br>
glibc-debuginfo-2.17-317.el7.x86_64 mar. 02 févr. 2021<br>
14:04:57 UTC<br>
glibc-debuginfo-common-2.17-317.el7.x86_64 mar. 02 févr. 2021<br>
14:04:53 UTC<br>
gpg-pubkey-b6792c39-53c4fbdd mar. 02 févr. 2021<br>
14:04:34 UTC<br>
tzdata-java-2021a-1.el7.noarch mer. 27 janv. 2021<br>
09:09:27 UTC<br>
tzdata-2021a-1.el7.noarch mer. 27 janv. 2021<br>
09:09:26 UTC<br>
sudo-1.8.23-10.el7_9.1.x86_64 mer. 27 janv. 2021<br>
09:09:26 UTC<br>
libtirpc-devel-0.2.4-0.16.el7.x86_64 mar. 26 janv. 2021<br>
12:53:45 UTC<br>
java-1.8.0-openjdk-1.8.0.282.b08-1.el7_9.x86_64 mar. 26 janv. 2021<br>
05:06:44 UTC<br>
<br>
We added libtirpc-devel on the 26/01.<br>
<br>
libtirpc-devel would, as the name imply, change something around the<br>
rpc subsystem.<br>
<br>
It happened around last week, when we started to notice the problem.<br>
<br>
It was not applied to 210, because 210 failed before it got to that<br>
point (since ansible stop as soon as the git update failed, and jenkins<br>
builder role is after the gluster-qa-script update).<br>
<br>
It was not applied to softserve provided VM either, so tests where<br>
working fine there. <br>
<br>
And indeed, once the package got removed, the tests were working again.<br>
<br>
Follow up<br>
=========<br>
<br>
So, I would like to know exactly what should be tested. Is gluster not<br>
compatible with libtirpc on C7 (as it work on C8), or is there some<br>
weird issue ? (cause from what I remember, RPC format is supposed to be<br>
compatible and covered by a specification)<br>
<br>
Should we test on C8 only ?<br>
<br>
<br>
-- <br>
Michael Scherer / He/Il/Er/Él<br>
Sysadmin, Community Infrastructure<br>
<br>
<br>
<br>
-------<br>
<br>
Community Meeting Calendar:<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><br>
<br>
</blockquote></div></div>