[Gluster-infra] Fwd: [Gluster-Maintainers] Build failed in Jenkins: regression-test-burn-in #4710

Atin Mukherjee amukherj at redhat.com
Tue Aug 27 02:33:28 UTC 2019 

Since last few days I was trying to understand the nightly failures we were
seeing even after addressing the port already in use issue. So here's the
analysis:

>From console output of
https://build.gluster.org/job/regression-test-burn-in/4710/consoleFull

<snip>

*19:51:56* Started by upstream project "nightly-master
<https://build.gluster.org/job/nightly-master/>" build number 843
<https://build.gluster.org/job/nightly-master/843>*19:51:56*
originally caused by:*19:51:56*  Started by timer*19:51:56* Running as
SYSTEM*19:51:57* Building remotely on builder209.aws.gluster.org
<https://build.gluster.org/computer/builder209.aws.gluster.org>
(centos7) in workspace
/home/jenkins/root/workspace/regression-test-burn-in*19:51:58* No
credentials specified*19:51:58*  > git rev-parse --is-inside-work-tree
# timeout=10*19:51:58* Fetching changes from the remote Git
repository*19:51:58*  > git config remote.origin.url
git://review.gluster.org/glusterfs.git # timeout=10*19:51:58* Fetching
upstream changes from git://review.gluster.org/glusterfs.git*19:51:58*
 > git --version # timeout=10*19:51:58*  > git fetch --tags --progress
git://review.gluster.org/glusterfs.git refs/heads/master #
timeout=10*19:52:01*  > git rev-parse origin/master^{commit} #
timeout=10*19:52:01* Checking out Revision
a31fad885c30cbc1bea652349c7d52bac1414c08 (origin/master)*19:52:01*  >
git config core.sparsecheckout # timeout=10*19:52:01  > git checkout
-f a31fad885c30cbc1bea652349c7d52bac1414c08 # timeout=10
19:52:02 Commit message: "tests: heal-info add --xml option for more
coverage"**19:52:02*  > git rev-list --no-walk
a31fad885c30cbc1bea652349c7d52bac1414c08 # timeout=10*19:52:02*
[regression-test-burn-in] $ /bin/bash
/tmp/jenkins7274529097702336737.sh*19:52:02* Start time Mon Aug 26
14:22:02 UTC 2019


</snip>

The latest commit which it picked up as part of git checkout is quite old
and hence we continue to see the similar failures in the latest nightly
runs which has been already addressed by commit c370c70

commit c370c70f77079339e2cfb7f284f3a2fb13fd2f97
Author: Mohit Agrawal <moagrawal at redhat.com>
Date:   Tue Aug 13 18:45:43 2019 +0530

    rpc: glusterd start is failed and throwing an error Address already in
use

    Problem: Some of the .t are failed due to bind is throwing
             an error EADDRINUSE

    Solution: After killing all gluster processes .t is trying
              to start glusterd but somehow if kernel has not cleaned
              up resources(socket) then glusterd startup is failed due to
              bind system call failure.To avoid the issue retries to call
              bind 10 times to execute system call succesfully

    Change-Id: Ia5fd6b788f7b211c1508c1b7304fc08a32266629
    Fixes: bz#1743020
    Signed-off-by: Mohit Agrawal <moagrawal at redhat.com>

So the (puzzling) question is - why are we picking up old commit?

In my local setup when I run the following command I do see the latest
commit id being picked up:

atin at dhcp35-96:~/codebase/upstream/glusterfs_master/glusterfs$ git
rev-parse origin/master^{commit} # timeout=10
7926992e65d0a07fdc784a6e45740306d9b4a9f2

atin at dhcp35-96:~/codebase/upstream/glusterfs_master/glusterfs$ git show
7926992e65d0a07fdc784a6e45740306d9b4a9f2
commit 7926992e65d0a07fdc784a6e45740306d9b4a9f2 (origin/master,
origin/HEAD, master)
Author: Sanju Rakonde <srakonde at redhat.com>
Date:   Mon Aug 26 12:38:40 2019 +0530

    glusterd: Unused value coverity fix

    CID: 1288765
    updates: bz#789278

    Change-Id: Ie6b01f81339769f44d82fd7c32ad0ed1a697c69c
    Signed-off-by: Sanju Rakonde <srakonde at redhat.com>



---------- Forwarded message ---------
From: <jenkins at build.gluster.org>
Date: Mon, Aug 26, 2019 at 11:32 PM
Subject: [Gluster-Maintainers] Build failed in Jenkins:
regression-test-burn-in #4710
To: <maintainers at gluster.org>


See <
https://build.gluster.org/job/regression-test-burn-in/4710/display/redirect>

------------------------------------------
[...truncated 4.18 MB...]
./tests/features/lock-migration/lkmigration-set-option.t  -  7 second
./tests/bugs/upcall/bug-1458127.t  -  7 second
./tests/bugs/transport/bug-873367.t  -  7 second
./tests/bugs/snapshot/bug-1260848.t  -  7 second
./tests/bugs/shard/shard-inode-refcount-test.t  -  7 second
./tests/bugs/replicate/bug-986905.t  -  7 second
./tests/bugs/replicate/bug-921231.t  -  7 second
./tests/bugs/replicate/bug-1132102.t  -  7 second
./tests/bugs/replicate/bug-1037501.t  -  7 second
./tests/bugs/posix/bug-1175711.t  -  7 second
./tests/bugs/posix/bug-1122028.t  -  7 second
./tests/bugs/glusterfs/bug-861015-log.t  -  7 second
./tests/bugs/fuse/bug-983477.t  -  7 second
./tests/bugs/ec/bug-1227869.t  -  7 second
./tests/bugs/distribute/bug-1086228.t  -  7 second
./tests/bugs/cli/bug-1087487.t  -  7 second
./tests/bitrot/br-stub.t  -  7 second
./tests/basic/ctime/ctime-noatime.t  -  7 second
./tests/basic/afr/ta-write-on-bad-brick.t  -  7 second
./tests/basic/afr/ta.t  -  7 second
./tests/basic/afr/ta-shd.t  -  7 second
./tests/basic/afr/root-squash-self-heal.t  -  7 second
./tests/basic/afr/granular-esh/add-brick.t  -  7 second
./tests/bugs/upcall/bug-1369430.t  -  6 second
./tests/bugs/snapshot/bug-1064768.t  -  6 second
./tests/bugs/shard/bug-1258334.t  -  6 second
./tests/bugs/replicate/bug-1250170-fsync.t  -  6 second
./tests/bugs/quota/bug-1250582-volume-reset-should-not-remove-quota-quota-deem-statfs.t
-  6 second
./tests/bugs/quota/bug-1243798.t  -  6 second
./tests/bugs/quota/bug-1104692.t  -  6 second
./tests/bugs/protocol/bug-1321578.t  -  6 second
./tests/bugs/nfs/bug-915280.t  -  6 second
./tests/bugs/io-cache/bug-858242.t  -  6 second
./tests/bugs/glusterfs-server/bug-877992.t  -  6 second
./tests/bugs/glusterfs/bug-902610.t  -  6 second
./tests/bugs/distribute/bug-884597.t  -  6 second
./tests/bugs/core/bug-1699025-brick-mux-detach-brick-fd-issue.t  -  6 second
./tests/bugs/core/bug-1168803-snapd-option-validation-fix.t  -  6 second
./tests/bugs/bug-1702299.t  -  6 second
./tests/bugs/bug-1371806_2.t  -  6 second
./tests/bugs/bug-1258069.t  -  6 second
./tests/bugs/bitrot/1209818-vol-info-show-scrub-process-properly.t  -  6
second
./tests/bugs/bitrot/1209751-bitrot-scrub-tunable-reset.t  -  6 second
./tests/basic/glusterd/thin-arbiter-volume-probe.t  -  6 second
./tests/basic/gfapi/libgfapi-fini-hang.t  -  6 second
./tests/basic/fencing/fencing-crash-conistency.t  -  6 second
./tests/basic/ec/statedump.t  -  6 second
./tests/basic/distribute/file-create.t  -  6 second
./tests/basic/afr/tarissue.t  -  6 second
./tests/basic/afr/gfid-heal.t  -  6 second
./tests/basic/afr/afr-read-hash-mode.t  -  6 second
./tests/basic/afr/add-brick-self-heal.t  -  6 second
./tests/gfid2path/gfid2path_fuse.t  -  5 second
./tests/bugs/shard/bug-1259651.t  -  5 second
./tests/bugs/replicate/bug-767585-gfid.t  -  5 second
./tests/bugs/replicate/bug-1686568-send-truncate-on-arbiter-from-shd.t  -
5 second
./tests/bugs/replicate/bug-1626994-info-split-brain.t  -  5 second
./tests/bugs/replicate/bug-1365455.t  -  5 second
./tests/bugs/replicate/bug-1101647.t  -  5 second
./tests/bugs/nfs/bug-877885.t  -  5 second
./tests/bugs/nfs/bug-847622.t  -  5 second
./tests/bugs/nfs/bug-1116503.t  -  5 second
./tests/bugs/md-cache/setxattr-prepoststat.t  -  5 second
./tests/bugs/md-cache/bug-1211863_unlink.t  -  5 second
./tests/bugs/md-cache/afr-stale-read.t  -  5 second
./tests/bugs/io-stats/bug-1598548.t  -  5 second
./tests/bugs/glusterfs/bug-895235.t  -  5 second
./tests/bugs/glusterfs/bug-856455.t  -  5 second
./tests/bugs/glusterfs/bug-848251.t  -  5 second
./tests/bugs/glusterd/quorum-value-check.t  -  5 second
./tests/bugs/glusterd/bug-1242875-do-not-pass-volinfo-quota.t  -  5 second
./tests/bugs/ec/bug-1179050.t  -  5 second
./tests/bugs/distribute/bug-912564.t  -  5 second
./tests/bugs/distribute/bug-1368012.t  -  5 second
./tests/bugs/core/bug-986429.t  -  5 second
./tests/bugs/core/bug-908146.t  -  5 second
./tests/bugs/bug-1371806_1.t  -  5 second
./tests/bugs/bitrot/bug-1229134-bitd-not-support-vol-set.t  -  5 second
./tests/bugs/bitrot/1207029-bitrot-daemon-should-start-on-valid-node.t  -
5 second
./tests/basic/playground/template-xlator-sanity.t  -  5 second
./tests/basic/hardlink-limit.t  -  5 second
./tests/basic/glusterd/arbiter-volume-probe.t  -  5 second
./tests/basic/ec/nfs.t  -  5 second
./tests/basic/ec/ec-read-policy.t  -  5 second
./tests/basic/ec/ec-anonymous-fd.t  -  5 second
./tests/basic/afr/arbiter-remove-brick.t  -  5 second
./tests/gfid2path/gfid2path_nfs.t  -  4 second
./tests/gfid2path/get-gfid-to-path.t  -  4 second
./tests/gfid2path/block-mount-access.t  -  4 second
./tests/bugs/upcall/bug-upcall-stat.t  -  4 second
./tests/bugs/trace/bug-797171.t  -  4 second
./tests/bugs/snapshot/bug-1178079.t  -  4 second
./tests/bugs/shard/bug-1342298.t  -  4 second
./tests/bugs/shard/bug-1272986.t  -  4 second
./tests/bugs/rpc/bug-954057.t  -  4 second
./tests/bugs/replicate/bug-886998.t  -  4 second
./tests/bugs/replicate/bug-1480525.t  -  4 second
./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t  -  4 second
./tests/bugs/replicate/bug-1325792.t  -  4 second
./tests/bugs/readdir-ahead/bug-1670253-consistent-metadata.t  -  4 second
./tests/bugs/posix/bug-gfid-path.t  -  4 second
./tests/bugs/posix/bug-765380.t  -  4 second
./tests/bugs/posix/bug-1619720.t  -  4 second
./tests/bugs/nfs/zero-atime.t  -  4 second
./tests/bugs/nfs/subdir-trailing-slash.t  -  4 second
./tests/bugs/nfs/socket-as-fifo.t  -  4 second
./tests/bugs/nfs/showmount-many-clients.t  -  4 second
./tests/bugs/nfs/bug-1161092-nfs-acls.t  -  4 second
./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t  -  4 second
./tests/bugs/glusterfs-server/bug-873549.t  -  4 second
./tests/bugs/glusterfs-server/bug-864222.t  -  4 second
./tests/bugs/glusterfs/bug-893378.t  -  4 second
./tests/bugs/glusterd/bug-948729/bug-948729-force.t  -  4 second
./tests/bugs/glusterd/bug-1482906-peer-file-blank-line.t  -  4 second
./tests/bugs/glusterd/bug-1091935-brick-order-check-from-cli-to-glusterd.t
-  4 second
./tests/bugs/geo-replication/bug-1296496.t  -  4 second
./tests/bugs/ec/bug-1161621.t  -  4 second
./tests/bugs/distribute/bug-1088231.t  -  4 second
./tests/bugs/cli/bug-977246.t  -  4 second
./tests/bugs/cli/bug-1004218.t  -  4 second
./tests/bugs/bug-1138841.t  -  4 second
./tests/bugs/bitrot/bug-1210684-scrub-pause-resume-error-handling.t  -  4
second
./tests/bugs/access-control/bug-1051896.t  -  4 second
./tests/bitrot/bug-1221914.t  -  4 second
./tests/basic/ec/ec-internal-xattrs.t  -  4 second
./tests/basic/distribute/non-root-unlink-stale-linkto.t  -  4 second
./tests/basic/distribute/bug-1265677-use-readdirp.t  -  4 second
./tests/basic/changelog/changelog-rename.t  -  4 second
./tests/basic/afr/ta-check-locks.t  -  4 second
./tests/basic/afr/heal-info.t  -  4 second
./tests/performance/quick-read.t  -  3 second
./tests/line-coverage/meta-max-coverage.t  -  3 second
./tests/bugs/upcall/bug-1422776.t  -  3 second
./tests/bugs/upcall/bug-1394131.t  -  3 second
./tests/bugs/unclassified/bug-1034085.t  -  3 second
./tests/bugs/snapshot/bug-1111041.t  -  3 second
./tests/bugs/shard/bug-1256580.t  -  3 second
./tests/bugs/shard/bug-1250855.t  -  3 second
./tests/bugs/replicate/bug-976800.t  -  3 second
./tests/bugs/replicate/bug-880898.t  -  3 second
./tests/bugs/read-only/bug-1134822-read-only-default-in-graph.t  -  3 second
./tests/bugs/readdir-ahead/bug-1446516.t  -  3 second
./tests/bugs/readdir-ahead/bug-1439640.t  -  3 second
./tests/bugs/readdir-ahead/bug-1390050.t  -  3 second
./tests/bugs/quota/bug-1287996.t  -  3 second
./tests/bugs/quick-read/bug-846240.t  -  3 second
./tests/bugs/nl-cache/bug-1451588.t  -  3 second
./tests/bugs/nfs/bug-1210338.t  -  3 second
./tests/bugs/nfs/bug-1166862.t  -  3 second
./tests/bugs/md-cache/bug-1632503.t  -  3 second
./tests/bugs/md-cache/bug-1476324.t  -  3 second
./tests/bugs/glusterfs-server/bug-861542.t  -  3 second
./tests/bugs/glusterfs/bug-869724.t  -  3 second
./tests/bugs/glusterfs/bug-844688.t  -  3 second
./tests/bugs/glusterfs/bug-1482528.t  -  3 second
./tests/bugs/glusterd/bug-948729/bug-948729.t  -  3 second
./tests/bugs/glusterd/bug-948729/bug-948729-mode-script.t  -  3 second
./tests/bugs/fuse/bug-1336818.t  -  3 second
./tests/bugs/fuse/bug-1126048.t  -  3 second
./tests/bugs/distribute/bug-907072.t  -  3 second
./tests/bugs/core/log-bug-1362520.t  -  3 second
./tests/bugs/core/io-stats-1322825.t  -  3 second
./tests/bugs/core/bug-913544.t  -  3 second
./tests/bugs/core/bug-845213.t  -  3 second
./tests/bugs/core/bug-834465.t  -  3 second
./tests/bugs/core/bug-1421721-mpx-toggle.t  -  3 second
./tests/bugs/core/bug-1135514-allow-setxattr-with-null-value.t  -  3 second
./tests/bugs/core/bug-1117951.t  -  3 second
./tests/bugs/core/949327.t  -  3 second
./tests/bugs/cli/bug-983317-volume-get.t  -  3 second
./tests/bugs/cli/bug-961307.t  -  3 second
./tests/bugs/access-control/bug-1387241.t  -  3 second
./tests/bitrot/bug-internal-xattrs-check-1243391.t  -  3 second
./tests/basic/quota-rename.t  -  3 second
./tests/basic/glusterd/check-cloudsync-ancestry.t  -  3 second
./tests/basic/fops-sanity.t  -  3 second
./tests/basic/fencing/test-fence-option.t  -  3 second
./tests/basic/ec/ec-fallocate.t  -  3 second
./tests/basic/ec/dht-rename.t  -  3 second
./tests/basic/distribute/lookup.t  -  3 second
./tests/basic/distribute/debug-xattrs.t  -  3 second
./tests/line-coverage/some-features-in-libglusterfs.t  -  2 second
./tests/bugs/unclassified/bug-991622.t  -  2 second
./tests/bugs/shard/bug-1245547.t  -  2 second
./tests/bugs/replicate/bug-884328.t  -  2 second
./tests/bugs/readdir-ahead/bug-1512437.t  -  2 second
./tests/bugs/posix/disallow-gfid-volumeid-removexattr.t  -  2 second
./tests/bugs/nfs/bug-970070.t  -  2 second
./tests/bugs/nfs/bug-1302948.t  -  2 second
./tests/bugs/logging/bug-823081.t  -  2 second
./tests/bugs/glusterfs-server/bug-889996.t  -  2 second
./tests/bugs/glusterfs/bug-860297.t  -  2 second
./tests/bugs/glusterfs/bug-811493.t  -  2 second
./tests/bugs/glusterd/bug-1085330-and-bug-916549.t  -  2 second
./tests/bugs/fuse/bug-1283103.t  -  2 second
./tests/bugs/distribute/bug-924265.t  -  2 second
./tests/bugs/distribute/bug-1204140.t  -  2 second
./tests/bugs/core/bug-924075.t  -  2 second
./tests/bugs/core/bug-903336.t  -  2 second
./tests/bugs/core/bug-1119582.t  -  2 second
./tests/bugs/core/bug-1111557.t  -  2 second
./tests/bugs/cli/bug-969193.t  -  2 second
./tests/bugs/cli/bug-949298.t  -  2 second
./tests/bugs/cli/bug-1378842-volume-get-all.t  -  2 second
./tests/basic/md-cache/bug-1418249.t  -  2 second
./tests/basic/afr/arbiter-cli.t  -  2 second
./tests/line-coverage/volfile-with-all-graph-syntax.t  -  1 second
./tests/bugs/shard/bug-1261773.t  -  1 second
./tests/bugs/replicate/ta-inode-refresh-read.t  -  1 second
./tests/bugs/glusterfs/bug-892730.t  -  1 second
./tests/bugs/glusterfs/bug-853690.t  -  1 second
./tests/bugs/cli/bug-921215.t  -  1 second
./tests/bugs/cli/bug-867252.t  -  1 second
./tests/bugs/cli/bug-764638.t  -  1 second
./tests/bugs/cli/bug-1047378.t  -  1 second
./tests/basic/posixonly.t  -  1 second
./tests/basic/peer-parsing.t  -  1 second
./tests/basic/netgroup_parsing.t  -  1 second
./tests/basic/gfapi/sink.t  -  1 second
./tests/basic/exports_parsing.t  -  1 second
./tests/basic/glusterfsd-args.t  -  0 second


4 test(s) failed
./tests/bugs/core/multiplex-limit-issue-151.t
./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t
./tests/bugs/glusterd/brick-mux-validation.t
./tests/bugs/glusterd/bug-1595320.t

0 test(s) generated core


10 test(s) needed retry
./tests/bugs/core/bug-1119582.t
./tests/bugs/core/multiplex-limit-issue-151.t
./tests/bugs/glusterd/add-brick-and-validate-replicated-volume-options.t
./tests/bugs/glusterd/brick-mux-validation.t
./tests/bugs/glusterd/bug-1595320.t
./tests/bugs/glusterd/bug-1696046.t
./tests/bugs/glusterd/optimized-basic-testcases.t
./tests/bugs/replicate/bug-1134691-afr-lookup-metadata-heal.t
./tests/bugs/replicate/bug-976800.t
./tests/bugs/snapshot/bug-1111041.t

Result is 1

tar: Removing leading `/' from member names
kernel.core_pattern = /%e-%p.core
Build step 'Execute shell' marked build as failure
_______________________________________________
maintainers mailing list
maintainers at gluster.org
https://lists.gluster.org/mailman/listinfo/maintainers
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-infra/attachments/20190827/f3c352c3/attachment-0001.html>

More information about the Gluster-infra mailing list