<div dir="ltr"><div>Since last few days I was trying to understand the nightly failures we were seeing even after addressing the port already in use issue. So here's the analysis:<br></div><div><br></div><div>From console output of <a href="https://build.gluster.org/job/regression-test-burn-in/4710/consoleFull">https://build.gluster.org/job/regression-test-burn-in/4710/consoleFull</a></div><div><br></div><div><snip></div><div><br></div><div><pre class="gmail-console-output"><span class="gmail-timestamp"><b>19:51:56</b> </span>Started by upstream project "<a href="https://build.gluster.org/job/nightly-master/" class="gmail-model-link">nightly-master</a>" build number <a href="https://build.gluster.org/job/nightly-master/843" class="gmail-model-link">843</a>
<span class="gmail-timestamp"><b>19:51:56</b> </span>originally caused by:
<span class="gmail-timestamp"><b>19:51:56</b> </span> Started by timer
<span class="gmail-timestamp"><b>19:51:56</b> </span>Running as SYSTEM
<span class="gmail-timestamp"><b>19:51:57</b> </span>Building remotely on <a href="https://build.gluster.org/computer/builder209.aws.gluster.org" class="gmail-model-link">builder209.aws.gluster.org</a> (centos7) in workspace /home/jenkins/root/workspace/regression-test-burn-in
<span class="gmail-timestamp"><b>19:51:58</b> </span>No credentials specified
<span class="gmail-timestamp"><b>19:51:58</b> </span> > git rev-parse --is-inside-work-tree # timeout=10
<span class="gmail-timestamp"><b>19:51:58</b> </span>Fetching changes from the remote Git repository
<span class="gmail-timestamp"><b>19:51:58</b> </span> > git config remote.origin.url git://<a href="http://review.gluster.org/glusterfs.git">review.gluster.org/glusterfs.git</a> # timeout=10
<span class="gmail-timestamp"><b>19:51:58</b> </span>Fetching upstream changes from git://<a href="http://review.gluster.org/glusterfs.git">review.gluster.org/glusterfs.git</a>
<span class="gmail-timestamp"><b>19:51:58</b> </span> > git --version # timeout=10
<span class="gmail-timestamp"><b>19:51:58</b> </span> > git fetch --tags --progress git://<a href="http://review.gluster.org/glusterfs.git">review.gluster.org/glusterfs.git</a> refs/heads/master # timeout=10
<span class="gmail-timestamp"><b>19:52:01</b> </span> > git rev-parse origin/master^{commit} # timeout=10
<span class="gmail-timestamp"><b>19:52:01</b> </span>Checking out Revision a31fad885c30cbc1bea652349c7d52bac1414c08 (origin/master)
<span class="gmail-timestamp"><b>19:52:01</b> </span> > git config core.sparsecheckout # timeout=10
<b><span class="gmail-timestamp">19:52:01 </span> > git checkout -f a31fad885c30cbc1bea652349c7d52bac1414c08 # timeout=10
<span class="gmail-timestamp">19:52:02 </span>Commit message: "tests: heal-info add --xml option for more coverage"</b>
<span class="gmail-timestamp"><b>19:52:02</b> </span> > git rev-list --no-walk a31fad885c30cbc1bea652349c7d52bac1414c08 # timeout=10
<span class="gmail-timestamp"><b>19:52:02</b> </span>[regression-test-burn-in] $ /bin/bash /tmp/jenkins7274529097702336737.sh
<span class="gmail-timestamp"><b>19:52:02</b> </span>Start time Mon Aug 26 14:22:02 UTC 2019</pre></div><div><br></div><div></snip><br></div><div><br></div><div>The latest commit which it picked up as part of git checkout is quite old and hence we continue to see the similar failures in the latest nightly runs which has been already addressed by commit c370c70</div><div><br></div><div>commit c370c70f77079339e2cfb7f284f3a2fb13fd2f97<br>Author: Mohit Agrawal <<a href="mailto:moagrawal@redhat.com">moagrawal@redhat.com</a>><br>Date: Tue Aug 13 18:45:43 2019 +0530<br><br> rpc: glusterd start is failed and throwing an error Address already in use<br> <br> Problem: Some of the .t are failed due to bind is throwing<br> an error EADDRINUSE<br> <br> Solution: After killing all gluster processes .t is trying<br> to start glusterd but somehow if kernel has not cleaned<br> up resources(socket) then glusterd startup is failed due to<br> bind system call failure.To avoid the issue retries to call<br> bind 10 times to execute system call succesfully<br> <br> Change-Id: Ia5fd6b788f7b211c1508c1b7304fc08a32266629<br> Fixes: bz#1743020<br> Signed-off-by: Mohit Agrawal <<a href="mailto:moagrawal@redhat.com">moagrawal@redhat.com</a>></div><div><br></div><div>So the (puzzling) question is - why are we picking up old commit?</div><div><br></div><div>In my local setup when I run the following command I do see the latest commit id being picked up:</div><div><br></div><div>atin@dhcp35-96:~/codebase/upstream/glusterfs_master/glusterfs$ git rev-parse origin/master^{commit} # timeout=10<br>7926992e65d0a07fdc784a6e45740306d9b4a9f2</div><div><br></div><div>atin@dhcp35-96:~/codebase/upstream/glusterfs_master/glusterfs$ git show 7926992e65d0a07fdc784a6e45740306d9b4a9f2<br>commit 7926992e65d0a07fdc784a6e45740306d9b4a9f2 (origin/master, origin/HEAD, master)<br>Author: Sanju Rakonde <<a href="mailto:srakonde@redhat.com">srakonde@redhat.com</a>><br>Date: Mon Aug 26 12:38:40 2019 +0530<br><br> glusterd: Unused value coverity fix<br> <br> CID: 1288765<br> updates: bz#789278<br> <br> Change-Id: Ie6b01f81339769f44d82fd7c32ad0ed1a697c69c<br> Signed-off-by: Sanju Rakonde <<a href="mailto:srakonde@redhat.com">srakonde@redhat.com</a>><br></div><div><br></div><div><br></div><div><br></div><div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br>From: <span dir="auto"><<a href="mailto:jenkins@build.gluster.org">jenkins@build.gluster.org</a>></span><br>Date: Mon, Aug 26, 2019 at 11:32 PM<br>Subject: [Gluster-Maintainers] Build failed in Jenkins: regression-test-burn-in #4710<br>To: <<a href="mailto:maintainers@gluster.org">maintainers@gluster.org</a>><br></div><br><br>See <<a href="https://build.gluster.org/job/regression-test-burn-in/4710/display/redirect" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-burn-in/4710/display/redirect</a>><br>
<br>
------------------------------------------<br>
[...truncated 4.18 MB...]<br>
./tests/features/lock-migration/lkmigration-set-option.t - 7 second<br>
./tests/bugs/upcall/bug-1458127.t - 7 second<br>
./tests/bugs/transport/bug-873367.t - 7 second<br>
./tests/bugs/snapshot/bug-1260848.t - 7 second<br>
./tests/bugs/shard/shard-inode-refcount-test.t - 7 second<br>
./tests/bugs/replicate/bug-986905.t - 7 second<br>
./tests/bugs/replicate/bug-921231.t - 7 second<br>
./tests/bugs/replicate/bug-1132102.t - 7 second<br>
./tests/bugs/replicate/bug-1037501.t - 7 second<br>
./tests/bugs/posix/bug-1175711.t - 7 second<br>
./tests/bugs/posix/bug-1122028.t - 7 second<br>
./tests/bugs/glusterfs/bug-861015-log.t - 7 second<br>
./tests/bugs/fuse/bug-983477.t - 7 second<br>
./tests/bugs/ec/bug-1227869.t - 7 second<br>
./tests/bugs/distribute/bug-1086228.t - 7 second<br>
./tests/bugs/cli/bug-1087487.t - 7 second<br>
./tests/bitrot/br-stub.t - 7 second<br>
./tests/basic/ctime/ctime-noatime.t - 7 second<br>
./tests/basic/afr/ta-write-on-bad-brick.t - 7 second<br>
./tests/basic/afr/ta.t - 7 second<br>
./tests/basic/afr/ta-shd.t - 7 second<br>
./tests/basic/afr/root-squash-self-heal.t - 7 second<br>
./tests/basic/afr/granular-esh/add-brick.t - 7 second<br>
./tests/bugs/upcall/bug-1369430.t - 6 second<br>
./tests/bugs/snapshot/bug-1064768.t - 6 second<br>
./tests/bugs/shard/bug-1258334.t - 6 second<br>
./tests/bugs/replicate/bug-1250170-fsync.t - 6 second<br>
./tests/bugs/quota/bug-1250582-volume-reset-should-not-remove-quota-quota-deem-statfs.t - 6 second<br>
./tests/bugs/quota/bug-1243798.t - 6 second<br>
./tests/bugs/quota/bug-1104692.t - 6 second<br>
./tests/bugs/protocol/bug-1321578.t - 6 second<br>
./tests/bugs/nfs/bug-915280.t - 6 second<br>
./tests/bugs/io-cache/bug-858242.t - 6 second<br>
./tests/bugs/glusterfs-server/bug-877992.t - 6 second<br>
./tests/bugs/glusterfs/bug-902610.t - 6 second<br>
./tests/bugs/distribute/bug-884597.t - 6 second<br>
./tests/bugs/core/bug-1699025-brick-mux-detach-brick-fd-issue.t - 6 second<br>
./tests/bugs/core/bug-1168803-snapd-option-validation-fix.t - 6 second<br>
./tests/bugs/bug-1702299.t - 6 second<br>
./tests/bugs/bug-1371806_2.t - 6 second<br>
./tests/bugs/bug-1258069.t - 6 second<br>
./tests/bugs/bitrot/1209818-vol-info-show-scrub-process-properly.t - 6 second<br>
./tests/bugs/bitrot/1209751-bitrot-scrub-tunable-reset.t - 6 second<br>
./tests/basic/glusterd/thin-arbiter-volume-probe.t - 6 second<br>
./tests/basic/gfapi/libgfapi-fini-hang.t - 6 second<br>
./tests/basic/fencing/fencing-crash-conistency.t - 6 second<br>
./tests/basic/ec/statedump.t - 6 second<br>
./tests/basic/distribute/file-create.t - 6 second<br>
./tests/basic/afr/tarissue.t - 6 second<br>
./tests/basic/afr/gfid-heal.t - 6 second<br>
./tests/basic/afr/afr-read-hash-mode.t - 6 second<br>
./tests/basic/afr/add-brick-self-heal.t - 6 second<br>
./tests/gfid2path/gfid2path_fuse.t - 5 second<br>
./tests/bugs/shard/bug-1259651.t - 5 second<br>
./tests/bugs/replicate/bug-767585-gfid.t - 5 second<br>
./tests/bugs/replicate/bug-1686568-send-truncate-on-arbiter-from-shd.t - 5 second<br>
./tests/bugs/replicate/bug-1626994-info-split-brain.t - 5 second<br>
./tests/bugs/replicate/bug-1365455.t - 5 second<br>
./tests/bugs/replicate/bug-1101647.t - 5 second<br>
./tests/bugs/nfs/bug-877885.t - 5 second<br>
./tests/bugs/nfs/bug-847622.t - 5 second<br>
./tests/bugs/nfs/bug-1116503.t - 5 second<br>
./tests/bugs/md-cache/setxattr-prepoststat.t - 5 second<br>
./tests/bugs/md-cache/bug-1211863_unlink.t - 5 second<br>
./tests/bugs/md-cache/afr-stale-read.t - 5 second<br>
./tests/bugs/io-stats/bug-1598548.t - 5 second<br>
./tests/bugs/glusterfs/bug-895235.t - 5 second<br>
./tests/bugs/glusterfs/bug-856455.t - 5 second<br>
./tests/bugs/glusterfs/bug-848251.t - 5 second<br>
./tests/bugs/glusterd/quorum-value-check.t - 5 second<br>
./tests/bugs/glusterd/bug-1242875-do-not-pass-volinfo-quota.t - 5 second<br>
./tests/bugs/ec/bug-1179050.t - 5 second<br>
./tests/bugs/distribute/bug-912564.t - 5 second<br>
./tests/bugs/distribute/bug-1368012.t - 5 second<br>
./tests/bugs/core/bug-986429.t - 5 second<br>
./tests/bugs/core/bug-908146.t - 5 second<br>
./tests/bugs/bug-1371806_1.t - 5 second<br>
./tests/bugs/bitrot/bug-1229134-bitd-not-support-vol-set.t - 5 second<br>
./tests/bugs/bitrot/1207029-bitrot-daemon-should-start-on-valid-node.t - 5 second<br>
./tests/basic/playground/template-xlator-sanity.t - 5 second<br>
./tests/basic/hardlink-limit.t - 5 second<br>
./tests/basic/glusterd/arbiter-volume-probe.t - 5 second<br>
./tests/basic/ec/nfs.t - 5 second<br>
./tests/basic/ec/ec-read-policy.t - 5 second<br>
./tests/basic/ec/ec-anonymous-fd.t - 5 second<br>
./tests/basic/afr/arbiter-remove-brick.t - 5 second<br>
./tests/gfid2path/gfid2path_nfs.t - 4 second<br>
./tests/gfid2path/get-gfid-to-path.t - 4 second<br>
./tests/gfid2path/block-mount-access.t - 4 second<br>
./tests/bugs/upcall/bug-upcall-stat.t - 4 second<br>
./tests/bugs/trace/bug-797171.t - 4 second<br>
./tests/bugs/snapshot/bug-1178079.t - 4 second<br>
./tests/bugs/shard/bug-1342298.t - 4 second<br>
./tests/bugs/shard/bug-1272986.t - 4 second<br>
./tests/bugs/rpc/bug-954057.t - 4 second<br>
./tests/bugs/replicate/bug-886998.t - 4 second<br>
./tests/bugs/replicate/bug-1480525.t - 4 second<br>
./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t - 4 second<br>
./tests/bugs/replicate/bug-1325792.t - 4 second<br>
./tests/bugs/readdir-ahead/bug-1670253-consistent-metadata.t - 4 second<br>
./tests/bugs/posix/bug-gfid-path.t - 4 second<br>
./tests/bugs/posix/bug-765380.t - 4 second<br>
./tests/bugs/posix/bug-1619720.t - 4 second<br>
./tests/bugs/nfs/zero-atime.t - 4 second<br>
./tests/bugs/nfs/subdir-trailing-slash.t - 4 second<br>
./tests/bugs/nfs/socket-as-fifo.t - 4 second<br>
./tests/bugs/nfs/showmount-many-clients.t - 4 second<br>
./tests/bugs/nfs/bug-1161092-nfs-acls.t - 4 second<br>
./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t - 4 second<br>
./tests/bugs/glusterfs-server/bug-873549.t - 4 second<br>
./tests/bugs/glusterfs-server/bug-864222.t - 4 second<br>
./tests/bugs/glusterfs/bug-893378.t - 4 second<br>
./tests/bugs/glusterd/bug-948729/bug-948729-force.t - 4 second<br>
./tests/bugs/glusterd/bug-1482906-peer-file-blank-line.t - 4 second<br>
./tests/bugs/glusterd/bug-1091935-brick-order-check-from-cli-to-glusterd.t - 4 second<br>
./tests/bugs/geo-replication/bug-1296496.t - 4 second<br>
./tests/bugs/ec/bug-1161621.t - 4 second<br>
./tests/bugs/distribute/bug-1088231.t - 4 second<br>
./tests/bugs/cli/bug-977246.t - 4 second<br>
./tests/bugs/cli/bug-1004218.t - 4 second<br>
./tests/bugs/bug-1138841.t - 4 second<br>
./tests/bugs/bitrot/bug-1210684-scrub-pause-resume-error-handling.t - 4 second<br>
./tests/bugs/access-control/bug-1051896.t - 4 second<br>
./tests/bitrot/bug-1221914.t - 4 second<br>
./tests/basic/ec/ec-internal-xattrs.t - 4 second<br>
./tests/basic/distribute/non-root-unlink-stale-linkto.t - 4 second<br>
./tests/basic/distribute/bug-1265677-use-readdirp.t - 4 second<br>
./tests/basic/changelog/changelog-rename.t - 4 second<br>
./tests/basic/afr/ta-check-locks.t - 4 second<br>
./tests/basic/afr/heal-info.t - 4 second<br>
./tests/performance/quick-read.t - 3 second<br>
./tests/line-coverage/meta-max-coverage.t - 3 second<br>
./tests/bugs/upcall/bug-1422776.t - 3 second<br>
./tests/bugs/upcall/bug-1394131.t - 3 second<br>
./tests/bugs/unclassified/bug-1034085.t - 3 second<br>
./tests/bugs/snapshot/bug-1111041.t - 3 second<br>
./tests/bugs/shard/bug-1256580.t - 3 second<br>
./tests/bugs/shard/bug-1250855.t - 3 second<br>
./tests/bugs/replicate/bug-976800.t - 3 second<br>
./tests/bugs/replicate/bug-880898.t - 3 second<br>
./tests/bugs/read-only/bug-1134822-read-only-default-in-graph.t - 3 second<br>
./tests/bugs/readdir-ahead/bug-1446516.t - 3 second<br>
./tests/bugs/readdir-ahead/bug-1439640.t - 3 second<br>
./tests/bugs/readdir-ahead/bug-1390050.t - 3 second<br>
./tests/bugs/quota/bug-1287996.t - 3 second<br>
./tests/bugs/quick-read/bug-846240.t - 3 second<br>
./tests/bugs/nl-cache/bug-1451588.t - 3 second<br>
./tests/bugs/nfs/bug-1210338.t - 3 second<br>
./tests/bugs/nfs/bug-1166862.t - 3 second<br>
./tests/bugs/md-cache/bug-1632503.t - 3 second<br>
./tests/bugs/md-cache/bug-1476324.t - 3 second<br>
./tests/bugs/glusterfs-server/bug-861542.t - 3 second<br>
./tests/bugs/glusterfs/bug-869724.t - 3 second<br>
./tests/bugs/glusterfs/bug-844688.t - 3 second<br>
./tests/bugs/glusterfs/bug-1482528.t - 3 second<br>
./tests/bugs/glusterd/bug-948729/bug-948729.t - 3 second<br>
./tests/bugs/glusterd/bug-948729/bug-948729-mode-script.t - 3 second<br>
./tests/bugs/fuse/bug-1336818.t - 3 second<br>
./tests/bugs/fuse/bug-1126048.t - 3 second<br>
./tests/bugs/distribute/bug-907072.t - 3 second<br>
./tests/bugs/core/log-bug-1362520.t - 3 second<br>
./tests/bugs/core/io-stats-1322825.t - 3 second<br>
./tests/bugs/core/bug-913544.t - 3 second<br>
./tests/bugs/core/bug-845213.t - 3 second<br>
./tests/bugs/core/bug-834465.t - 3 second<br>
./tests/bugs/core/bug-1421721-mpx-toggle.t - 3 second<br>
./tests/bugs/core/bug-1135514-allow-setxattr-with-null-value.t - 3 second<br>
./tests/bugs/core/bug-1117951.t - 3 second<br>
./tests/bugs/core/949327.t - 3 second<br>
./tests/bugs/cli/bug-983317-volume-get.t - 3 second<br>
./tests/bugs/cli/bug-961307.t - 3 second<br>
./tests/bugs/access-control/bug-1387241.t - 3 second<br>
./tests/bitrot/bug-internal-xattrs-check-1243391.t - 3 second<br>
./tests/basic/quota-rename.t - 3 second<br>
./tests/basic/glusterd/check-cloudsync-ancestry.t - 3 second<br>
./tests/basic/fops-sanity.t - 3 second<br>
./tests/basic/fencing/test-fence-option.t - 3 second<br>
./tests/basic/ec/ec-fallocate.t - 3 second<br>
./tests/basic/ec/dht-rename.t - 3 second<br>
./tests/basic/distribute/lookup.t - 3 second<br>
./tests/basic/distribute/debug-xattrs.t - 3 second<br>
./tests/line-coverage/some-features-in-libglusterfs.t - 2 second<br>
./tests/bugs/unclassified/bug-991622.t - 2 second<br>
./tests/bugs/shard/bug-1245547.t - 2 second<br>
./tests/bugs/replicate/bug-884328.t - 2 second<br>
./tests/bugs/readdir-ahead/bug-1512437.t - 2 second<br>
./tests/bugs/posix/disallow-gfid-volumeid-removexattr.t - 2 second<br>
./tests/bugs/nfs/bug-970070.t - 2 second<br>
./tests/bugs/nfs/bug-1302948.t - 2 second<br>
./tests/bugs/logging/bug-823081.t - 2 second<br>
./tests/bugs/glusterfs-server/bug-889996.t - 2 second<br>
./tests/bugs/glusterfs/bug-860297.t - 2 second<br>
./tests/bugs/glusterfs/bug-811493.t - 2 second<br>
./tests/bugs/glusterd/bug-1085330-and-bug-916549.t - 2 second<br>
./tests/bugs/fuse/bug-1283103.t - 2 second<br>
./tests/bugs/distribute/bug-924265.t - 2 second<br>
./tests/bugs/distribute/bug-1204140.t - 2 second<br>
./tests/bugs/core/bug-924075.t - 2 second<br>
./tests/bugs/core/bug-903336.t - 2 second<br>
./tests/bugs/core/bug-1119582.t - 2 second<br>
./tests/bugs/core/bug-1111557.t - 2 second<br>
./tests/bugs/cli/bug-969193.t - 2 second<br>
./tests/bugs/cli/bug-949298.t - 2 second<br>
./tests/bugs/cli/bug-1378842-volume-get-all.t - 2 second<br>
./tests/basic/md-cache/bug-1418249.t - 2 second<br>
./tests/basic/afr/arbiter-cli.t - 2 second<br>
./tests/line-coverage/volfile-with-all-graph-syntax.t - 1 second<br>
./tests/bugs/shard/bug-1261773.t - 1 second<br>
./tests/bugs/replicate/ta-inode-refresh-read.t - 1 second<br>
./tests/bugs/glusterfs/bug-892730.t - 1 second<br>
./tests/bugs/glusterfs/bug-853690.t - 1 second<br>
./tests/bugs/cli/bug-921215.t - 1 second<br>
./tests/bugs/cli/bug-867252.t - 1 second<br>
./tests/bugs/cli/bug-764638.t - 1 second<br>
./tests/bugs/cli/bug-1047378.t - 1 second<br>
./tests/basic/posixonly.t - 1 second<br>
./tests/basic/peer-parsing.t - 1 second<br>
./tests/basic/netgroup_parsing.t - 1 second<br>
./tests/basic/gfapi/sink.t - 1 second<br>
./tests/basic/exports_parsing.t - 1 second<br>
./tests/basic/glusterfsd-args.t - 0 second<br>
<br>
<br>
4 test(s) failed <br>
./tests/bugs/core/multiplex-limit-issue-151.t<br>
./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t<br>
./tests/bugs/glusterd/brick-mux-validation.t<br>
./tests/bugs/glusterd/bug-1595320.t<br>
<br>
0 test(s) generated core <br>
<br>
<br>
10 test(s) needed retry <br>
./tests/bugs/core/bug-1119582.t<br>
./tests/bugs/core/multiplex-limit-issue-151.t<br>
./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t<br>
./tests/bugs/glusterd/brick-mux-validation.t<br>
./tests/bugs/glusterd/bug-1595320.t<br>
./tests/bugs/glusterd/bug-1696046.t<br>
./tests/bugs/glusterd/optimized-basic-testcases.t<br>
./tests/bugs/replicate/bug-1134691-afr-lookup-metadata-heal.t<br>
./tests/bugs/replicate/bug-976800.t<br>
./tests/bugs/snapshot/bug-1111041.t<br>
<br>
Result is 1<br>
<br>
tar: Removing leading `/' from member names<br>
kernel.core_pattern = /%e-%p.core<br>
Build step 'Execute shell' marked build as failure<br>
_______________________________________________<br>
maintainers mailing list<br>
<a href="mailto:maintainers@gluster.org" target="_blank">maintainers@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/maintainers</a><br>
</div></div></div>