<div dir="ltr"><div>I recently added 3 builders builder208, builder209, builder210 to the regression pool. Network to these new builders did not come up because it was looking for non-existing ethernet card eth0 on reboot and hence failing. I'll reconnect them back and update here once I fix the issue today.</div><div><br></div><div>Sorry for the inconvenience.<br></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jun 4, 2019 at 7:07 PM Yaniv Kaul <<a href="mailto:ykaul@redhat.com">ykaul@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">What was the result of this investigation? I suspect seeing the same issue on builder209[1].</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Y.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">[1] <a href="https://build.gluster.org/job/centos7-regression/6302/consoleFull" target="_blank">https://build.gluster.org/job/centos7-regression/6302/consoleFull</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Apr 5, 2019 at 5:40 PM Michael Scherer <<a href="mailto:mscherer@redhat.com" target="_blank">mscherer@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Le vendredi 05 avril 2019 à 16:55 +0530, Nithya Balachandran a écrit :<br>
> On Fri, 5 Apr 2019 at 12:16, Michael Scherer <<a href="mailto:mscherer@redhat.com" target="_blank">mscherer@redhat.com</a>><br>
> wrote:<br>
> <br>
> > Le jeudi 04 avril 2019 à 18:24 +0200, Michael Scherer a écrit :<br>
> > > Le jeudi 04 avril 2019 à 19:10 +0300, Yaniv Kaul a écrit :<br>
> > > > I'm not convinced this is solved. Just had what I believe is a<br>
> > > > similar<br>
> > > > failure:<br>
> > > > <br>
> > > > *00:12:02.532* A dependency job for rpc-statd.service failed.<br>
> > > > See<br>
> > > > 'journalctl -xe' for details.*00:12:02.532* mount.nfs:<br>
> > > > rpc.statd is<br>
> > > > not running but is required for remote locking.*00:12:02.532*<br>
> > > > mount.nfs: Either use '-o nolock' to keep locks local, or start<br>
> > > > statd.*00:12:02.532* mount.nfs: an incorrect mount option was<br>
> > > > specified<br>
> > > > <br>
> > > > (of course, it can always be my patch!)<br>
> > > > <br>
> > > > <a href="https://build.gluster.org/job/centos7-regression/5384/console" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos7-regression/5384/console</a><br>
> > > <br>
> > > same issue, different builder (206). I will check them all, as<br>
> > > the<br>
> > > issue is more widespread than I expected (or it did popup since<br>
> > > last<br>
> > > time I checked).<br>
> > <br>
> > Deepshika did notice that the issue came back on one server<br>
> > (builder202) after a reboot, so the rpcbind issue is not related to<br>
> > the<br>
> > network initscript one, so the RCA continue.<br>
> > <br>
> > We are looking for another workaround involving fiddling with the<br>
> > socket (until we find why it do use ipv6 at boot, but not after,<br>
> > when<br>
> > ipv6 is disabled).<br>
> > <br>
> <br>
> Could this be relevant?<br>
> <a href="https://access.redhat.com/solutions/2798411" rel="noreferrer" target="_blank">https://access.redhat.com/solutions/2798411</a><br>
<br>
Good catch.<br>
<br>
So, we already do that, Nigel took care of that (after 2 days of<br>
research). But I didn't knew the exact symptoms, and decided to double<br>
check just in case.<br>
<br>
And... there is no sysctl.conf in the initrd. Running dracut -v -f do<br>
not change anything.<br>
<br>
Running "dracut -v -f -H" take care of that (and this fix the problem),<br>
but:<br>
- our ansible script already run that<br>
- -H is hostonly, which is already the default on EL7 according to the<br>
doc. <br>
<br>
However, if dracut-config-generic is installed, it doesn't build a<br>
hostonly initrd, and so do not include the sysctl.conf file (who break<br>
rpcbnd, who break the test suite).<br>
<br>
And for some reason, it is installed the image in ec2 (likely default),<br>
but not by default on the builders.<br>
<br>
So what happen is that after a kernel upgrade, dracut rebuild a generic<br>
initrd instead of a hostonly one, who break things. And kernel was<br>
likely upgraded recently (and upgrade happen nightly (for some value of<br>
"night"), so we didn't see that earlier, nor with a fresh system.<br>
<br>
<br>
So now, we have several solution:<br>
- be explicit on using hostonly in dracut, so this doesn't happen again<br>
(or not for this reason)<br>
<br>
- disable ipv6 in rpcbind in a cleaner way (to be tested)<br>
<br>
- get the test suite work with ip v6<br>
<br>
In the long term, I also want to monitor the processes, but for that, I<br>
need a VPN between the nagios server and ec2, and that project got<br>
blocked by several issues (like EC2 not support ecdsa keys, and we use<br>
that for ansible, so we have to come back to RSA for full automated<br>
deployment, and openvon requires to use certificates, so I need a newer<br>
python openssl for doing what I want, and RHEL 7 is too old, etc, etc).<br>
<br>
As the weekend approach for me, I just rebuilt the initrd for the time<br>
being. I guess forcing hostonly is the safest fix for now, but this<br>
will be for monday.<br>
-- <br>
Michael Scherer<br>
Sysadmin, Community Infrastructure and Platform, OSAS<br>
<br>
<br>
</blockquote></div>
_______________________________________________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/836554017" rel="noreferrer" target="_blank">https://bluejeans.com/836554017</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/486278655" rel="noreferrer" target="_blank">https://bluejeans.com/486278655</a><br>
<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><br>
<br>
</blockquote></div>