[Gluster-devel] Update on georep failure

Tue Feb 2 18:14:14 UTC 2021

Hi,

so we finally found the cause of the georep failure, after several days
of work from Deepshika and I. 

Short story:
============

side effect of adding libtirpc-devel on EL 7:
https://github.com/gluster/project-infrastructure/issues/115

Long story:
===========

So we first puzzled on why it was failing just on some builders and not
others, especially since it was working fine on softserve VMs. 

We tried to look for the usual suspect, rebooted, reinstalled, searched
if there was something weird (too much ssh keys, not enough inode, some
hardware issue), but nothing obvious. 

After trying to find my way in the logs file and a few weird leads
(like, why gsyncd was running gcc ? (answer: ctypes)), I was left with
a rather cryptic message:

[2021-02-02 15:19:00.040817 +0000] I
[socket.c:929:__socket_server_bind] 0-socket.gfchangelog: closing
(AF_UNIX) reuse check socket 18
[2021-02-02 15:19:02.041641 +0000] W [xdr-
rpcclnt.c:68:rpc_request_to_xdr] 0-rpc: failed to encode call msg
[2021-02-02 15:19:02.041673 +0000] E [rpc-
clnt.c:1507:rpc_clnt_record_build_record] 0-gfchangelog: Failed to
build record header
[2021-02-02 15:19:02.041683 +0000] W [rpc-clnt.c:1664:rpc_clnt_submit]
0-gfchangelog: cannot build rpc-record
[2021-02-02 15:19:02.041692 +0000] E [MSGID: 132023] [gf-
changelog.c:285:gf_changelog_setup_rpc] 0-gfchangelog: Could not
initiate probe RPC, bailing out!!! 
[2021-02-02 15:19:02.041809 +0000] E [MSGID: 132022] [gf-
changelog.c:583:gf_changelog_register_generic] 0-gfchangelog: Error
registering with changelog xlator 

Given that all gluster is around RPC, it would be unlikely that rpc is
broken, but that's the only messages we had.

We also found that the only builder that was working was builder 210.
Upon looking, we found that 210 failed to be updated with ansible, due
to some debugging we forgot to revert, which made this task fail: 
https://github.com/gluster/gluster.org_ansible_configuration/blob/master/roles/gluster_qa_scripts/tasks/main.yml#L7

But it wasn't clear how that would change anything, since the only diff
was a "set -e" that wasn't removed.

Then Deepshika started to test more than georep, and she noticed that a
lot of others tests were failing, with the same exact message about
rpc. 

And she started to wonder if anything was recently changed. And indeed:

# rpm -qa --last | head -n 15
yum-plugin-auto-update-debug-info-1.1.31-54.el7_8.noarch mar. 02 févr.
2021 14:04:59 UTC
python3-debuginfo-3.6.8-18.el7.x86_64         mar. 02 févr. 2021
14:04:58 UTC
glibc-debuginfo-2.17-317.el7.x86_64           mar. 02 févr. 2021
14:04:57 UTC
glibc-debuginfo-common-2.17-317.el7.x86_64    mar. 02 févr. 2021
14:04:53 UTC
gpg-pubkey-b6792c39-53c4fbdd                  mar. 02 févr. 2021
14:04:34 UTC
tzdata-java-2021a-1.el7.noarch                mer. 27 janv. 2021
09:09:27 UTC
tzdata-2021a-1.el7.noarch                     mer. 27 janv. 2021
09:09:26 UTC
sudo-1.8.23-10.el7_9.1.x86_64                 mer. 27 janv. 2021
09:09:26 UTC
libtirpc-devel-0.2.4-0.16.el7.x86_64          mar. 26 janv. 2021
12:53:45 UTC
java-1.8.0-openjdk-1.8.0.282.b08-1.el7_9.x86_64 mar. 26 janv. 2021
05:06:44 UTC

We added libtirpc-devel on the 26/01.

libtirpc-devel would, as the name imply, change something around the
rpc subsystem.

It happened around last week, when we started to notice the problem.

It was not applied to 210, because 210 failed before it got to that
point (since ansible stop as soon as the git update failed, and jenkins
builder role is after the gluster-qa-script update).

It was not applied to softserve provided VM either, so tests where
working fine there. 

And indeed, once the package got removed, the tests were working again.

Follow up
=========

So, I would like to know exactly what should be tested. Is gluster not
compatible with libtirpc on C7 (as it work on C8), or is there some
weird issue ? (cause from what I remember, RPC format is supposed to be
compatible and covered by a specification)

Should we test on C8 only ?

-- 
Michael Scherer / He/Il/Er/Él
Sysadmin, Community Infrastructure

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20210202/c9dfbf2c/attachment.sig>