[Gluster-Maintainers] Build failed in Jenkins: regression-test-with-multiplex #1611
Sunil Kumar Heggodu Gopala Acharya
sheggodu at redhat.com
Thu Jan 16 08:53:45 UTC 2020
Hi,
Please have the log location fixed so that others can take look into the
failures.
https://build.gluster.org/job/regression-test-with-multiplex/1611/consoleFull
*21:10:48* echo "Logs archived in http://${SERVER}/${filename}"
Regards,
Sunil kumar Acharya
Associate manager, Software Engineering
Red Hat
<https://www.redhat.com>
T: +91-8067935170 <http://redhatemailsignature-marketing.itos.redhat.com/>
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
On Thu, Jan 16, 2020 at 2:18 PM Sanju Rakonde <srakonde at redhat.com> wrote:
> The below glusterd test cases are constantly failing in brick-mux
> regression:
> ./tests/bugs/glusterd/bug-857330/normal.t
> ./tests/bugs/glusterd/bug-857330/xml.t
> ./tests/bugs/glusterd/quorum-validation.t
>
> ./tests/bugs/glusterd/bug-857330/normal.t and
> ./tests/bugs/glusterd/bug-857330/xml.t are timed-out after 200 seconds. I
> don't find any abnormality in the logs. we need to increase the time(?).
> I'm unable to run these tests in my local setup as they are always failing
> saying "ModuleNotFoundError: No module named 'xattr' " Is the same
> happening in CI as well?
>
> Also, we don't print the output of "prove -vf <test>" when the test gets
> timed-out. It will be great if we print the output. It will help us to
> debug and to check which step took more time.
>
> ./tests/bugs/glusterd/quorum-validation.t is failing because of the
> regression caused by https://review.gluster.org/#/c/glusterfs/+/21651/.
> Rafi is looking into this issue. To explain the issue in brief, "after a
> reboot, glusterd is spawning multiple brick processes for a single brick
> instance and volume status shows the brick as offline".
>
> I have gone through the logs and I see that glusterd is sending a detach
> request for brick2 once the brick2 is attached successfully to the existing
> brick process of brick1 and spawning a new brick process for it. I also
> cross-checked it by looking at the "ps -ax | grep glusterfsd" output.
>
>
> Status of volume: patchy
> Gluster process TCP Port RDMA Port Online
> Pid
>
> ------------------------------------------------------------------------------
> Brick 127.1.1.1:/d/backends/1/patchy0 49153 0 Y
> 28492
> Brick 127.1.1.2:/d/backends/2/patchy1 N/A N/A N
> N/A
> Brick 127.1.1.2:/d/backends/2/patchy2 N/A N/A N
> N/A
>
> Task Status of Volume patchy
>
> ------------------------------------------------------------------------------
> There are no active volume tasks
>
> 28492 ? Ssl 0:00 /usr/local/sbin/glusterfsd -s 127.1.1.1
> --volfile-id patchy.127.1.1.1.d-backends-1-patchy0 -p
> /d/backends/1/run/gluster/vols/patchy/127.1.1.1-d-backends-1-patchy0.pid -S
> /var/run/gluster/f96de9a0bebaefd6.socket --brick-name /d/backends/1/patchy0
> -l /var/log/glusterfs/1/bricks/d-backends-1-patchy0.log --xlator-option
> *-posix.glusterd-uuid=32807d18-ac26-44b1-8cf4-9cf072b13f35 --process-name
> brick --brick-port 49153 --xlator-option patchy-server.listen-port=49153
> --xlator-option transport.socket.bind-address=127.1.1.1 --brick-mux
> 28529 ? Ssl 0:00 /usr/local/sbin/glusterfsd -s 127.1.1.2
> --volfile-id patchy.127.1.1.2.d-backends-2-patchy1 -p
> /d/backends/2/run/gluster/vols/patchy/127.1.1.2-d-backends-2-patchy1.pid -S
> /var/run/gluster/0df89e2f99a00139.socket --brick-name /d/backends/2/patchy1
> -l /var/log/glusterfs/2/bricks/d-backends-2-patchy1.log --xlator-option
> *-posix.glusterd-uuid=4e538359-d7fb-437a-a222-0047ff323976 --process-name
> brick --brick-port 49154 --xlator-option patchy-server.listen-port=49154
> --xlator-option transport.socket.bind-address=127.1.1.2 --brick-mux
> 28547 ? Ssl 0:00 /usr/local/sbin/glusterfsd -s 127.1.1.2
> --volfile-id patchy.127.1.1.2.d-backends-2-patchy2 -p
> /d/backends/2/run/gluster/vols/patchy/127.1.1.2-d-backends-2-patchy2.pid -S
> /var/run/gluster/a4b51a62080fd909.socket --brick-name /d/backends/2/patchy2
> -l /var/log/glusterfs/2/bricks/d-backends-2-patchy2.log --xlator-option
> *-posix.glusterd-uuid=4e538359-d7fb-437a-a222-0047ff323976 --process-name
> brick --brick-port 49155 --xlator-option patchy-server.listen-port=49155
> --xlator-option transport.socket.bind-address=127.1.1.2 --brick-mux
> 28558 ? Ssl 0:00 /usr/local/sbin/glusterfsd -s 127.1.1.2
> --volfile-id patchy.127.1.1.2.d-backends-2-patchy2 -p
> /d/backends/2/run/gluster/vols/patchy/127.1.1.2-d-backends-2-patchy2.pid -S
> /var/run/gluster/a4b51a62080fd909.socket --brick-name /d/backends/2/patchy2
> -l /var/log/glusterfs/2/bricks/d-backends-2-patchy2.log --xlator-option
> *-posix.glusterd-uuid=4e538359-d7fb-437a-a222-0047ff323976 --process-name
> brick --brick-port 49156 --xlator-option patchy-server.listen-port=49156
> --xlator-option transport.socket.bind-address=127.1.1.2 --brick-mux
> 28605 pts/6 S+ 0:00 grep glusterfsd
>
>
> On Wed, Jan 15, 2020 at 2:40 AM <jenkins at build.gluster.org> wrote:
>
>> See <
>> https://build.gluster.org/job/regression-test-with-multiplex/1611/display/redirect?page=changes
>> >
>>
>> Changes:
>>
>> [Xavi Hernandez] xlators/storage: remove duplicated includes
>>
>> [Amar Tumballi] xlator/bit-rot-stub-helpers: structure logging
>>
>> [Shwetha K Acharya] tools/glusterfind: Remove an extra argument
>>
>>
>> ------------------------------------------
>> [...truncated 3.51 MB...]
>> ./tests/bugs/upcall/bug-1458127.t - 14 second
>> ./tests/bugs/snapshot/bug-1250387.t - 14 second
>> ./tests/bugs/snapshot/bug-1202436-calculate-quota-cksum-during-snap-restore.t
>> - 14 second
>> ./tests/bugs/snapshot/bug-1064768.t - 14 second
>> ./tests/bugs/shard/bug-1272986.t - 14 second
>> ./tests/bugs/replicate/bug-976800.t - 14 second
>> ./tests/bugs/replicate/bug-1101647.t - 14 second
>> ./tests/bugs/readdir-ahead/bug-1446516.t - 14 second
>> ./tests/bugs/quota/bug-1287996.t - 14 second
>> ./tests/bugs/nfs/bug-877885.t - 14 second
>> ./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t - 14 second
>> ./tests/bugs/glusterfs-server/bug-877992.t - 14 second
>> ./tests/bugs/glusterd/bug-1482906-peer-file-blank-line.t - 14 second
>> ./tests/bugs/geo-replication/bug-1296496.t - 14 second
>> ./tests/bugs/bitrot/1209818-vol-info-show-scrub-process-properly.t - 14
>> second
>> ./tests/basic/volfile-sanity.t - 14 second
>> ./tests/basic/ctime/ctime-mdata-legacy-files.t - 14 second
>> ./tests/basic/afr/heal-quota.t - 14 second
>> ./tests/basic/afr/data-self-heal.t - 14 second
>> ./tests/basic/afr/arbiter-statfs.t - 14 second
>> ./tests/line-coverage/cli-volume-top-profile-coverage.t - 13 second
>> ./tests/line-coverage/afr-heal-info.t - 13 second
>> ./tests/bugs/transport/bug-873367.t - 13 second
>> ./tests/bugs/snapshot/bug-1260848.t - 13 second
>> ./tests/bugs/replicate/bug-880898.t - 13 second
>> ./tests/bugs/replicate/bug-1365455.t - 13 second
>> ./tests/bugs/read-only/bug-1134822-read-only-default-in-graph.t - 13
>> second
>> ./tests/bugs/readdir-ahead/bug-1512437.t - 13 second
>> ./tests/bugs/posix/bug-1619720.t - 13 second
>> ./tests/bugs/nl-cache/bug-1451588.t - 13 second
>> ./tests/bugs/nfs/bug-1166862.t - 13 second
>> ./tests/bugs/glusterfs-server/bug-864222.t - 13 second
>> ./tests/bugs/glusterfs/bug-856455.t - 13 second
>> ./tests/bugs/distribute/bug-1204140.t - 13 second
>> ./tests/bugs/distribute/bug-1086228.t - 13 second
>> ./tests/bugs/core/io-stats-1322825.t - 13 second
>> ./tests/bugs/core/bug-913544.t - 13 second
>> ./tests/bugs/cli/bug-1113476.t - 13 second
>> ./tests/bugs/changelog/bug-1208470.t - 13 second
>> ./tests/bugs/bitrot/1209751-bitrot-scrub-tunable-reset.t - 13 second
>> ./tests/basic/xlator-pass-through-sanity.t - 13 second
>> ./tests/basic/multiplex.t - 13 second
>> ./tests/basic/md-cache/bug-1317785.t - 13 second
>> ./tests/basic/glusterd/volume-brick-count.t - 13 second
>> ./tests/basic/glusterd/volfile_server_switch.t - 13 second
>> ./tests/basic/gfid-access.t - 13 second
>> ./tests/basic/gfapi/glfs_sysrq.t - 13 second
>> ./tests/basic/gfapi/gfapi-trunc.t - 13 second
>> ./tests/basic/gfapi/gfapi-load-volfile.t - 13 second
>> ./tests/basic/gfapi/gfapi-async-calls-test.t - 13 second
>> ./tests/basic/ec/quota.t - 13 second
>> ./tests/basic/afr/resolve.t - 13 second
>> ./tests/basic/afr/read-subvol-entry.t - 13 second
>> ./tests/basic/afr/arbiter-add-brick.t - 13 second
>> ./tests/basic/afr/afr-up.t - 13 second
>> ./tests/features/readdir-ahead.t - 12 second
>> ./tests/bugs/upcall/bug-1422776.t - 12 second
>> ./tests/bugs/shard/bug-1256580.t - 12 second
>> ./tests/bugs/replicate/bug-1480525.t - 12 second
>> ./tests/bugs/readdir-ahead/bug-1390050.t - 12 second
>> ./tests/bugs/quick-read/bug-846240.t - 12 second
>> ./tests/bugs/nfs/bug-1210338.t - 12 second
>> ./tests/bugs/nfs/bug-1161092-nfs-acls.t - 12 second
>> ./tests/bugs/md-cache/bug-1476324.t - 12 second
>> ./tests/bugs/glusterfs/bug-893378.t - 12 second
>> ./tests/bugs/fuse/bug-1336818.t - 12 second
>> ./tests/bugs/distribute/bug-907072.t - 12 second
>> ./tests/bugs/core/bug-1119582.t - 12 second
>> ./tests/bugs/bitrot/bug-1210684-scrub-pause-resume-error-handling.t -
>> 12 second
>> ./tests/basic/ios-dump.t - 12 second
>> ./tests/basic/gfapi/gfapi-statx-basic.t - 12 second
>> ./tests/basic/gfapi/bug1613098.t - 12 second
>> ./tests/basic/ec/ec-rebalance.t - 12 second
>> ./tests/basic/distribute/brick-down.t - 12 second
>> ./tests/basic/ctime/ctime-readdir.t - 12 second
>> ./tests/basic/cdc.t - 12 second
>> ./tests/basic/afr/read-subvol-data.t - 12 second
>> ./tests/basic/afr/granular-esh/granular-esh.t - 12 second
>> ./tests/line-coverage/cli-peer-and-volume-operations.t - 11 second
>> ./tests/gfid2path/get-gfid-to-path.t - 11 second
>> ./tests/bugs/snapshot/bug-1178079.t - 11 second
>> ./tests/bugs/nfs/socket-as-fifo.t - 11 second
>> ./tests/bugs/nfs/showmount-many-clients.t - 11 second
>> ./tests/bugs/nfs/bug-847622.t - 11 second
>> ./tests/bugs/glusterfs-server/bug-873549.t - 11 second
>> ./tests/bugs/glusterd/bug-1085330-and-bug-916549.t - 11 second
>> ./tests/bugs/fuse/bug-1283103.t - 11 second
>> ./tests/bugs/cli/bug-977246.t - 11 second
>> ./tests/bugs/cli/bug-969193.t - 11 second
>> ./tests/bugs/cli/bug-1087487.t - 11 second
>> ./tests/bugs/bitrot/1209752-volume-status-should-show-bitrot-scrub-info.t
>> - 11 second
>> ./tests/bugs/access-control/bug-1051896.t - 11 second
>> ./tests/basic/quota-nfs.t - 11 second
>> ./tests/basic/global-threading.t - 11 second
>> ./tests/basic/ec/ec-up.t - 11 second
>> ./tests/basic/ctime/ctime-ec-rebalance.t - 11 second
>> ./tests/basic/afr/split-brain-open.t - 11 second
>> ./tests/performance/quick-read.t - 10 second
>> ./tests/gfid2path/block-mount-access.t - 10 second
>> ./tests/bugs/upcall/bug-1394131.t - 10 second
>> ./tests/bugs/nfs/zero-atime.t - 10 second
>> ./tests/bugs/nfs/subdir-trailing-slash.t - 10 second
>> ./tests/bugs/glusterfs-server/bug-861542.t - 10 second
>> ./tests/bugs/glusterfs/bug-860297.t - 10 second
>> ./tests/bugs/glusterfs/bug-844688.t - 10 second
>> ./tests/bugs/core/bug-834465.t - 10 second
>> ./tests/bugs/core/bug-1168803-snapd-option-validation-fix.t - 10 second
>> ./tests/bugs/cli/bug-1004218.t - 10 second
>> ./tests/bugs/access-control/bug-1387241.t - 10 second
>> ./tests/bitrot/bug-1221914.t - 10 second
>> ./tests/basic/distribute/throttle-rebal.t - 10 second
>> ./tests/basic/ctime/ctime-rep-rebalance.t - 10 second
>> ./tests/basic/ctime/ctime-rep-heal.t - 10 second
>> ./tests/basic/afr/granular-esh/replace-brick.t - 10 second
>> ./tests/basic/afr/granular-esh/granular-indices-but-non-granular-heal.t
>> - 10 second
>> ./tests/basic/afr/gfid-mismatch.t - 10 second
>> ./tests/basic/afr/client-side-heal.t - 10 second
>> ./tests/bugs/upcall/bug-upcall-stat.t - 9 second
>> ./tests/bugs/posix/disallow-gfid-volumeid-removexattr.t - 9 second
>> ./tests/bugs/logging/bug-823081.t - 9 second
>> ./tests/bugs/glusterfs-server/bug-889996.t - 9 second
>> ./tests/bugs/core/log-bug-1362520.t - 9 second
>> ./tests/bugs/core/bug-1421721-mpx-toggle.t - 9 second
>> ./tests/bugs/core/949327.t - 9 second
>> ./tests/bugs/cli/bug-983317-volume-get.t - 9 second
>> ./tests/bugs/bug-1138841.t - 9 second
>> ./tests/basic/hardlink-limit.t - 9 second
>> ./tests/basic/fop-sampling.t - 9 second
>> ./tests/basic/ctime/ctime-ec-heal.t - 9 second
>> ./tests/basic/afr/stale-file-lookup.t - 9 second
>> ./tests/basic/afr/root-squash-self-heal.t - 9 second
>> ./tests/basic/afr/replace-brick-self-heal.t - 9 second
>> ./tests/bugs/snapshot/bug-1111041.t - 8 second
>> ./tests/bugs/replicate/bug-884328.t - 8 second
>> ./tests/bugs/nfs/bug-1302948.t - 8 second
>> ./tests/bugs/core/bug-924075.t - 8 second
>> ./tests/bugs/core/bug-845213.t - 8 second
>> ./tests/bugs/core/bug-1135514-allow-setxattr-with-null-value.t - 8
>> second
>> ./tests/bugs/core/bug-1117951.t - 8 second
>> ./tests/bugs/cli/bug-961307.t - 8 second
>> ./tests/bitrot/bug-internal-xattrs-check-1243391.t - 8 second
>> ./tests/basic/gfapi/libgfapi-fini-hang.t - 8 second
>> ./tests/basic/ec/ec-root-heal.t - 8 second
>> ./tests/basic/distribute/file-create.t - 8 second
>> ./tests/basic/ctime/ctime-noatime.t - 8 second
>> ./tests/basic/afr/ta-write-on-bad-brick.t - 8 second
>> ./tests/basic/afr/ta-shd.t - 8 second
>> ./tests/basic/afr/tarissue.t - 8 second
>> ./tests/basic/afr/ta-read.t - 8 second
>> ./tests/basic/afr/add-brick-self-heal.t - 8 second
>> ./tests/gfid2path/gfid2path_nfs.t - 7 second
>> ./tests/gfid2path/gfid2path_fuse.t - 7 second
>> ./tests/bugs/unclassified/bug-1034085.t - 7 second
>> ./tests/bugs/shard/bug-1261773.t - 7 second
>> ./tests/bugs/glusterfs/bug-811493.t - 7 second
>> ./tests/bugs/fuse/bug-1126048.t - 7 second
>> ./tests/bugs/distribute/bug-924265.t - 7 second
>> ./tests/bugs/core/bug-908146.t - 7 second
>> ./tests/bugs/bitrot/1207029-bitrot-daemon-should-start-on-valid-node.t
>> - 7 second
>> ./tests/basic/quota-rename.t - 7 second
>> ./tests/basic/glusterd/thin-arbiter-volume-probe.t - 7 second
>> ./tests/basic/fencing/fencing-crash-conistency.t - 7 second
>> ./tests/basic/ec/statedump.t - 7 second
>> ./tests/basic/ec/nfs.t - 7 second
>> ./tests/basic/distribute/file-rename.t - 7 second
>> ./tests/basic/afr/ta.t - 7 second
>> ./tests/basic/afr/granular-esh/add-brick.t - 7 second
>> ./tests/basic/afr/gfid-heal.t - 7 second
>> ./tests/line-coverage/meta-max-coverage.t - 6 second
>> ./tests/bugs/trace/bug-797171.t - 6 second
>> ./tests/bugs/nfs/bug-970070.t - 6 second
>> ./tests/bugs/glusterfs/bug-853690.t - 6 second
>> ./tests/bugs/core/bug-986429.t - 6 second
>> ./tests/bugs/core/bug-1111557.t - 6 second
>> ./tests/bugs/cli/bug-1047378.t - 6 second
>> ./tests/basic/playground/template-xlator-sanity.t - 6 second
>> ./tests/basic/glusterd/arbiter-volume-probe.t - 6 second
>> ./tests/basic/ec/ec-read-policy.t - 6 second
>> ./tests/basic/ec/ec-anonymous-fd.t - 6 second
>> ./tests/basic/ctime/ctime-utimesat.t - 6 second
>> ./tests/basic/changelog/changelog-rename.t - 6 second
>> ./tests/basic/afr/heal-info.t - 6 second
>> ./tests/basic/afr/arbiter-remove-brick.t - 6 second
>> ./tests/basic/afr/afr-read-hash-mode.t - 6 second
>> ./tests/line-coverage/some-features-in-libglusterfs.t - 5 second
>> ./tests/bugs/unclassified/bug-991622.t - 5 second
>> ./tests/bugs/core/bug-903336.t - 5 second
>> ./tests/bugs/cli/bug-921215.t - 5 second
>> ./tests/bugs/cli/bug-1378842-volume-get-all.t - 5 second
>> ./tests/basic/ec/dht-rename.t - 5 second
>> ./tests/basic/distribute/non-root-unlink-stale-linkto.t - 5 second
>> ./tests/basic/distribute/bug-1265677-use-readdirp.t - 5 second
>> ./tests/bugs/replicate/ta-inode-refresh-read.t - 4 second
>> ./tests/basic/glusterd/check-cloudsync-ancestry.t - 4 second
>> ./tests/basic/ec/ec-internal-xattrs.t - 4 second
>> ./tests/basic/ec/ec-fallocate.t - 4 second
>> ./tests/basic/distribute/lookup.t - 4 second
>> ./tests/basic/afr/ta-check-locks.t - 4 second
>> ./tests/bugs/cli/bug-867252.t - 3 second
>> ./tests/bugs/cli/bug-764638.t - 3 second
>> ./tests/basic/fops-sanity.t - 3 second
>> ./tests/basic/fencing/test-fence-option.t - 3 second
>> ./tests/basic/distribute/debug-xattrs.t - 3 second
>> ./tests/bugs/glusterfs/bug-892730.t - 2 second
>> ./tests/basic/peer-parsing.t - 2 second
>> ./tests/basic/md-cache/bug-1418249.t - 2 second
>> ./tests/basic/afr/arbiter-cli.t - 2 second
>> ./tests/line-coverage/volfile-with-all-graph-syntax.t - 1 second
>> ./tests/bugs/cli/bug-949298.t - 1 second
>> ./tests/basic/netgroup_parsing.t - 1 second
>> ./tests/basic/gfapi/sink.t - 1 second
>> ./tests/basic/exports_parsing.t - 1 second
>> ./tests/basic/posixonly.t - 0 second
>> ./tests/basic/glusterfsd-args.t - 0 second
>>
>>
>> 9 test(s) failed
>> ./tests/basic/fencing/afr-lock-heal-advanced.t
>> ./tests/basic/fencing/afr-lock-heal-basic.t
>> ./tests/bugs/distribute/bug-1117851.t
>> ./tests/bugs/glusterd/bug-857330/normal.t
>> ./tests/bugs/glusterd/bug-857330/xml.t
>> ./tests/bugs/glusterd/quorum-validation.t
>> ./tests/bugs/glusterd/rebalance-operations-in-single-node.t
>> ./tests/bugs/index/bug-1559004-EMLINK-handling.t
>> ./tests/bugs/rpc/bug-884452.t
>>
>> 0 test(s) generated core
>>
>>
>> 12 test(s) needed retry
>> ./tests/basic/fencing/afr-lock-heal-advanced.t
>> ./tests/basic/fencing/afr-lock-heal-basic.t
>> ./tests/basic/glusterd/thin-arbiter-volume-probe.t
>> ./tests/bugs/distribute/bug-1117851.t
>> ./tests/bugs/distribute/bug-1125824.t
>> ./tests/bugs/glusterd/bug-857330/normal.t
>> ./tests/bugs/glusterd/bug-857330/xml.t
>> ./tests/bugs/glusterd/quorum-validation.t
>> ./tests/bugs/glusterd/rebalance-operations-in-single-node.t
>> ./tests/bugs/index/bug-1559004-EMLINK-handling.t
>> ./tests/bugs/rpc/bug-884452.t
>> ./tests/bugs/snapshot/bug-1597662.t
>>
>> Result is 124
>>
>> tar: Removing leading `/' from member names
>> kernel.core_pattern = /%e-%p.core
>> Build step 'Execute shell' marked build as failure
>>
>>
>
> --
> Thanks,
> Sanju
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/maintainers/attachments/20200116/1761d43e/attachment-0001.html>
More information about the maintainers
mailing list