<div><div dir="ltr"><div>I took a quick look at the builders and noticed both have the same error of 'Cannot allocate memory' which comes up every time when the builder is rebooted after a build abort. It is happening in the same pattern. Though there's no such memory consumption on the builders. </div><div dir="auto"><br></div><div dir="auto">I’m investigating more on this.</div></div></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 9, 2019 at 10:02 AM Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 8, 2019 at 7:38 PM Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">builder204 needs to be fixed, too many failures, mostly none of the patches are passing regression.<br></div></blockquote><div><br></div><div>And with that builder201 joins the pool, <a href="https://build.gluster.org/job/centos7-regression/5943/consoleFull" target="_blank">https://build.gluster.org/job/centos7-regression/5943/consoleFull</a><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 8, 2019 at 9:53 AM Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 8, 2019 at 7:16 AM Sanju Rakonde <<a href="mailto:srakonde@redhat.com" target="_blank">srakonde@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Deepshikha,<div><br></div><div>I see the failure here[1] which ran on builder206. So, we are good.</div></div></div></blockquote><div><br></div><div>Not really, <a href="https://build.gluster.org/job/centos7-regression/5909/consoleFull" target="_blank">https://build.gluster.org/job/centos7-regression/5909/consoleFull</a> failed on builder204 for similar reasons I believe?</div><div><br></div><div>I am bit more worried on this issue being resurfacing more often these days. What can we do to fix this permanently?</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div><br></div><div>[1] <a href="https://build.gluster.org/job/centos7-regression/5901/consoleFull" target="_blank">https://build.gluster.org/job/centos7-regression/5901/consoleFull</a></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 8, 2019 at 12:23 AM Deepshikha Khandelwal <<a href="mailto:dkhandel@redhat.com" target="_blank">dkhandel@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Sanju, can you please give us more info about the failures. <br></div><div><br></div><div>I see the failures occurring on just one of the builder (builder206). I'm taking it back offline for now. <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 7, 2019 at 9:42 PM Michael Scherer <<a href="mailto:mscherer@redhat.com" target="_blank">mscherer@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Le mardi 07 mai 2019 à 20:04 +0530, Sanju Rakonde a écrit :<br>
> Looks like is_nfs_export_available started failing again in recent<br>
> centos-regressions.<br>
> <br>
> Michael, can you please check?<br>
<br>
I will try but I am leaving for vacation tonight, so if I find nothing,<br>
until I leave, I guess Deepshika will have to look.<br>
<br>
> On Wed, Apr 24, 2019 at 5:30 PM Yaniv Kaul <<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>> wrote:<br>
> <br>
> > <br>
> > <br>
> > On Tue, Apr 23, 2019 at 5:15 PM Michael Scherer <<br>
> > <a href="mailto:mscherer@redhat.com" target="_blank">mscherer@redhat.com</a>><br>
> > wrote:<br>
> > <br>
> > > Le lundi 22 avril 2019 à 22:57 +0530, Atin Mukherjee a écrit :<br>
> > > > Is this back again? The recent patches are failing regression<br>
> > > > :-\ .<br>
> > > <br>
> > > So, on builder206, it took me a while to find that the issue is<br>
> > > that<br>
> > > nfs (the service) was running.<br>
> > > <br>
> > > ./tests/basic/afr/tarissue.t failed, because the nfs<br>
> > > initialisation<br>
> > > failed with a rather cryptic message:<br>
> > > <br>
> > > [2019-04-23 13:17:05.371733] I<br>
> > > [socket.c:991:__socket_server_bind] 0-<br>
> > > socket.nfs-server: process started listening on port (38465)<br>
> > > [2019-04-23 13:17:05.385819] E<br>
> > > [socket.c:972:__socket_server_bind] 0-<br>
> > > socket.nfs-server: binding to failed: Address already in use<br>
> > > [2019-04-23 13:17:05.385843] E<br>
> > > [socket.c:974:__socket_server_bind] 0-<br>
> > > socket.nfs-server: Port is already in use<br>
> > > [2019-04-23 13:17:05.385852] E [socket.c:3788:socket_listen] 0-<br>
> > > socket.nfs-server: __socket_server_bind failed;closing socket 14<br>
> > > <br>
> > > I found where this came from, but a few stuff did surprised me:<br>
> > > <br>
> > > - the order of print is different that the order in the code<br>
> > > <br>
> > <br>
> > Indeed strange...<br>
> > <br>
> > > - the message on "started listening" didn't take in account the<br>
> > > fact<br>
> > > that bind failed on:<br>
> > > <br>
> > <br>
> > Shouldn't it bail out if it failed to bind?<br>
> > Some missing 'goto out' around line 975/976?<br>
> > Y.<br>
> > <br>
> > > <br>
> > > <br>
> > > <br>
> > > <br>
<a href="https://github.com/gluster/glusterfs/blob/master/rpc/rpc-transport/socket/src/socket.c#L967" rel="noreferrer" target="_blank">https://github.com/gluster/glusterfs/blob/master/rpc/rpc-transport/socket/src/socket.c#L967</a><br>
> > > <br>
> > > The message about port 38465 also threw me off the track. The<br>
> > > real<br>
> > > issue is that the service nfs was already running, and I couldn't<br>
> > > find<br>
> > > anything listening on port 38465<br>
> > > <br>
> > > once I do service nfs stop, it no longer failed.<br>
> > > <br>
> > > So far, I do know why nfs.service was activated.<br>
> > > <br>
> > > But at least, 206 should be fixed, and we know a bit more on what<br>
> > > would<br>
> > > be causing some failure.<br>
> > > <br>
> > > <br>
> > > <br>
> > > > On Wed, 3 Apr 2019 at 19:26, Michael Scherer <<br>
> > > > <a href="mailto:mscherer@redhat.com" target="_blank">mscherer@redhat.com</a>><br>
> > > > wrote:<br>
> > > > <br>
> > > > > Le mercredi 03 avril 2019 à 16:30 +0530, Atin Mukherjee a<br>
> > > > > écrit :<br>
> > > > > > On Wed, Apr 3, 2019 at 11:56 AM Jiffin Thottan <<br>
> > > > > > <a href="mailto:jthottan@redhat.com" target="_blank">jthottan@redhat.com</a>><br>
> > > > > > wrote:<br>
> > > > > > <br>
> > > > > > > Hi,<br>
> > > > > > > <br>
> > > > > > > is_nfs_export_available is just a wrapper around<br>
> > > > > > > "showmount"<br>
> > > > > > > command AFAIR.<br>
> > > > > > > I saw following messages in console output.<br>
> > > > > > > mount.nfs: rpc.statd is not running but is required for<br>
> > > > > > > remote<br>
> > > > > > > locking.<br>
> > > > > > > 05:06:55 mount.nfs: Either use '-o nolock' to keep locks<br>
> > > > > > > local,<br>
> > > > > > > or<br>
> > > > > > > start<br>
> > > > > > > statd.<br>
> > > > > > > 05:06:55 mount.nfs: an incorrect mount option was<br>
> > > > > > > specified<br>
> > > > > > > <br>
> > > > > > > For me it looks rpcbind may not be running on the<br>
> > > > > > > machine.<br>
> > > > > > > Usually rpcbind starts automatically on machines, don't<br>
> > > > > > > know<br>
> > > > > > > whether it<br>
> > > > > > > can happen or not.<br>
> > > > > > > <br>
> > > > > > <br>
> > > > > > That's precisely what the question is. Why suddenly we're<br>
> > > > > > seeing<br>
> > > > > > this<br>
> > > > > > happening too frequently. Today I saw atleast 4 to 5 such<br>
> > > > > > failures<br>
> > > > > > already.<br>
> > > > > > <br>
> > > > > > Deepshika - Can you please help in inspecting this?<br>
> > > > > <br>
> > > > > So we think (we are not sure) that the issue is a bit<br>
> > > > > complex.<br>
> > > > > <br>
> > > > > What we were investigating was nightly run fail on aws. When<br>
> > > > > the<br>
> > > > > build<br>
> > > > > crash, the builder is restarted, since that's the easiest way<br>
> > > > > to<br>
> > > > > clean<br>
> > > > > everything (since even with a perfect test suite that would<br>
> > > > > clean<br>
> > > > > itself, we could always end in a corrupt state on the system,<br>
> > > > > WRT<br>
> > > > > mount, fs, etc).<br>
> > > > > <br>
> > > > > In turn, this seems to cause trouble on aws, since cloud-init <br>
> > > > > or<br>
> > > > > something rename eth0 interface to ens5, without cleaning to<br>
> > > > > the<br>
> > > > > network configuration.<br>
> > > > > <br>
> > > > > So the network init script fail (because the image say "start<br>
> > > > > eth0"<br>
> > > > > and<br>
> > > > > that's not present), but fail in a weird way. Network is<br>
> > > > > initialised<br>
> > > > > and working (we can connect), but the dhclient process is not<br>
> > > > > in<br>
> > > > > the<br>
> > > > > right cgroup, and network.service is in failed state.<br>
> > > > > Restarting<br>
> > > > > network didn't work. In turn, this mean that rpc-statd refuse<br>
> > > > > to<br>
> > > > > start<br>
> > > > > (due to systemd dependencies), which seems to impact various<br>
> > > > > NFS<br>
> > > > > tests.<br>
> > > > > <br>
> > > > > We have also seen that on some builders, rpcbind pick some IP<br>
> > > > > v6<br>
> > > > > autoconfiguration, but we can't reproduce that, and there is<br>
> > > > > no ip<br>
> > > > > v6<br>
> > > > > set up anywhere. I suspect the network.service failure is<br>
> > > > > somehow<br>
> > > > > involved, but fail to see how. In turn, rpcbind.socket not<br>
> > > > > starting<br>
> > > > > could cause NFS test troubles.<br>
> > > > > <br>
> > > > > Our current stop gap fix was to fix all the builders one by<br>
> > > > > one.<br>
> > > > > Remove<br>
> > > > > the config, kill the rogue dhclient, restart network service.<br>
> > > > > <br>
> > > > > However, we can't be sure this is going to fix the problem<br>
> > > > > long<br>
> > > > > term<br>
> > > > > since this only manifest after a crash of the test suite, and<br>
> > > > > it<br>
> > > > > doesn't happen so often. (plus, it was working before some<br>
> > > > > day in<br>
> > > > > the<br>
> > > > > past, when something did make this fail, and I do not know if<br>
> > > > > that's a<br>
> > > > > system upgrade, or a test change, or both).<br>
> > > > > <br>
> > > > > So we are still looking at it to have a complete<br>
> > > > > understanding of<br>
> > > > > the<br>
> > > > > issue, but so far, we hacked our way to make it work (or so<br>
> > > > > do I<br>
> > > > > think).<br>
> > > > > <br>
> > > > > Deepshika is working to fix it long term, by fixing the issue<br>
> > > > > regarding<br>
> > > > > eth0/ens5 with a new base image.<br>
> > > > > --<br>
> > > > > Michael Scherer<br>
> > > > > Sysadmin, Community Infrastructure and Platform, OSAS<br>
> > > > > <br>
> > > > > <br>
> > > > > --<br>
> > > > <br>
> > > > - Atin (atinm)<br>
> > > <br>
> > > --<br>
> > > Michael Scherer<br>
> > > Sysadmin, Community Infrastructure<br>
> > > <br>
> > > <br>
> > > <br>
> > > _______________________________________________<br>
> > > Gluster-devel mailing list<br>
> > > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > > <a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><br>
> > <br>
> > _______________________________________________<br>
> > Gluster-devel mailing list<br>
> > <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> > <a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><br>
> <br>
> <br>
> <br>
-- <br>
Michael Scherer<br>
Sysadmin, Community Infrastructure<br>
<br>
<br>
<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a></blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="m_1813258206301134145gmail-m_-408774628014834163gmail-m_5184492975264877617gmail-m_-6533020112101670479gmail-m_1731302203545716766gmail-m_4461305160343800102gmail-m_3559919090678772387gmail_signature"><div dir="ltr"><div>Thanks,<br></div>Sanju<br></div></div>
_______________________________________________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/836554017" rel="noreferrer" target="_blank">https://bluejeans.com/836554017</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/486278655" rel="noreferrer" target="_blank">https://bluejeans.com/486278655</a><br>
<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><br>
<br>
</blockquote></div></div></div>
</blockquote></div>
</blockquote></div></div></div>
</blockquote></div>
</div>