<div><div dir="auto">Is this back again? The recent patches are failing regression :-\ .</div></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 3 Apr 2019 at 19:26, Michael Scherer <<a href="mailto:mscherer@redhat.com">mscherer@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Le mercredi 03 avril 2019 à 16:30 +0530, Atin Mukherjee a écrit :<br>
> On Wed, Apr 3, 2019 at 11:56 AM Jiffin Thottan <<a href="mailto:jthottan@redhat.com" target="_blank">jthottan@redhat.com</a>><br>
> wrote:<br>
> <br>
> > Hi,<br>
> > <br>
> > is_nfs_export_available is just a wrapper around "showmount"<br>
> > command AFAIR.<br>
> > I saw following messages in console output.<br>
> > mount.nfs: rpc.statd is not running but is required for remote<br>
> > locking.<br>
> > 05:06:55 mount.nfs: Either use '-o nolock' to keep locks local, or<br>
> > start<br>
> > statd.<br>
> > 05:06:55 mount.nfs: an incorrect mount option was specified<br>
> > <br>
> > For me it looks rpcbind may not be running on the machine.<br>
> > Usually rpcbind starts automatically on machines, don't know<br>
> > whether it<br>
> > can happen or not.<br>
> > <br>
> <br>
> That's precisely what the question is. Why suddenly we're seeing this<br>
> happening too frequently. Today I saw atleast 4 to 5 such failures<br>
> already.<br>
> <br>
> Deepshika - Can you please help in inspecting this?<br>
<br>
So we think (we are not sure) that the issue is a bit complex.<br>
<br>
What we were investigating was nightly run fail on aws. When the build<br>
crash, the builder is restarted, since that's the easiest way to clean<br>
everything (since even with a perfect test suite that would clean<br>
itself, we could always end in a corrupt state on the system, WRT<br>
mount, fs, etc).<br>
<br>
In turn, this seems to cause trouble on aws, since cloud-init or<br>
something rename eth0 interface to ens5, without cleaning to the<br>
network configuration. <br>
<br>
So the network init script fail (because the image say "start eth0" and<br>
that's not present), but fail in a weird way. Network is initialised<br>
and working (we can connect), but the dhclient process is not in the<br>
right cgroup, and network.service is in failed state. Restarting<br>
network didn't work. In turn, this mean that rpc-statd refuse to start<br>
(due to systemd dependencies), which seems to impact various NFS tests.<br>
<br>
We have also seen that on some builders, rpcbind pick some IP v6<br>
autoconfiguration, but we can't reproduce that, and there is no ip v6<br>
set up anywhere. I suspect the network.service failure is somehow<br>
involved, but fail to see how. In turn, rpcbind.socket not starting<br>
could cause NFS test troubles.<br>
<br>
Our current stop gap fix was to fix all the builders one by one. Remove<br>
the config, kill the rogue dhclient, restart network service. <br>
<br>
However, we can't be sure this is going to fix the problem long term<br>
since this only manifest after a crash of the test suite, and it<br>
doesn't happen so often. (plus, it was working before some day in the<br>
past, when something did make this fail, and I do not know if that's a<br>
system upgrade, or a test change, or both).<br>
<br>
So we are still looking at it to have a complete understanding of the<br>
issue, but so far, we hacked our way to make it work (or so do I<br>
think).<br>
<br>
Deepshika is working to fix it long term, by fixing the issue regarding<br>
eth0/ens5 with a new base image.<br>
-- <br>
Michael Scherer<br>
Sysadmin, Community Infrastructure and Platform, OSAS<br>
<br>
<br>
</blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">- Atin (atinm)</div>