<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 7, 2019 at 10:07 AM Amar Tumballi Suryanarayan <<a href="mailto:atumball@redhat.com">atumball@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Got time to test subdir-mount.t failing in brick-mux scenario.<div><br></div><div>I noticed some issues, where I need further help from glusterd team.</div><div><br></div><div>subdir-mount.t expects 'hook' script to run after add-brick to make sure the required subdirectories are healed and are present in new bricks. This is important as subdir mount expects the subdirs to exist for successful mount.</div><div><br></div><div>But in case of brick-mux setup, I see that in some cases (6/10), hook script (add-brick/post-hook/S13-create-subdir-mount.sh) started getting executed after 20second of finishing the add-brick command. Due to this, the mount which we execute after add-brick failed.</div><div><br></div><div>My question is, what is making post hook script to run so late ??</div></div></blockquote><div><br></div><div>It's not only the add-brick in the post hook. Given post hook scripts are async in nature, I see the respective hook scripts of create/start/set volume operation have executed quite a late which is very surprising until and unless some thread has been stuck for quite a while. Unfortunately for both Mohit and I, the issue isn't reproducible locally. Mohit would give it a try in softserve infra but at this point of time, there's no conclusive evidence, the analysis continues.</div><div><br></div><div>Amar - would it be possible for you to do a git blame given you can reproduce this? May 31 nightly (<a href="https://build.gluster.org/job/regression-test-with-multiplex/1359/">https://build.gluster.org/job/regression-test-with-multiplex/1359/</a>) is when this test started failing.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>I can recreate the issues locally on my laptop too. </div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jun 1, 2019 at 4:55 PM Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="auto">subdir-mount.t has started failing in brick mux regression nightly. This needs to be fixed.</div></div><div dir="auto"><br></div><div dir="auto">Raghavendra - did we manage to get any further clue on uss.t failure?</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br>From: <span dir="ltr"><<a href="mailto:jenkins@build.gluster.org" target="_blank">jenkins@build.gluster.org</a>></span><br>Date: Fri, 31 May 2019 at 23:34<br>Subject: [Gluster-Maintainers] Build failed in Jenkins: regression-test-with-multiplex #1359<br>To: <<a href="mailto:maintainers@gluster.org" target="_blank">maintainers@gluster.org</a>>, <<a href="mailto:rkavunga@redhat.com" target="_blank">rkavunga@redhat.com</a>>, <<a href="mailto:amarts@redhat.com" target="_blank">amarts@redhat.com</a>>, <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>>, <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>><br></div><br><br>See <<a href="https://build.gluster.org/job/regression-test-with-multiplex/1359/display/redirect?page=changes" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/1359/display/redirect?page=changes</a>><br>
<br>
Changes:<br>
<br>
[atin] glusterd: add an op-version check<br>
<br>
[atin] glusterd/svc: glusterd_svcs_stop should call individual wrapper function<br>
<br>
[atin] glusterd/svc: Stop stale process using the glusterd_proc_stop<br>
<br>
[Amar Tumballi] lcov: more coverage to shard, old-protocol, sdfs<br>
<br>
[Kotresh H R] tests/geo-rep: Add EC volume test case<br>
<br>
[Amar Tumballi] glusterfsd/cleanup: Protect graph object under a lock<br>
<br>
[Mohammed Rafi KC] glusterd/shd: Optimize the glustershd manager to send reconfigure<br>
<br>
[Kotresh H R] tests/geo-rep: Add tests to cover glusterd geo-rep<br>
<br>
[atin] glusterd: Optimize code to copy dictionary in handshake code path<br>
<br>
------------------------------------------<br>
[...truncated 3.18 MB...]<br>
./tests/basic/afr/stale-file-lookup.t - 9 second<br>
./tests/basic/afr/granular-esh/replace-brick.t - 9 second<br>
./tests/basic/afr/granular-esh/add-brick.t - 9 second<br>
./tests/basic/afr/gfid-mismatch.t - 9 second<br>
./tests/performance/open-behind.t - 8 second<br>
./tests/features/ssl-authz.t - 8 second<br>
./tests/features/readdir-ahead.t - 8 second<br>
./tests/bugs/upcall/bug-1458127.t - 8 second<br>
./tests/bugs/transport/bug-873367.t - 8 second<br>
./tests/bugs/replicate/bug-1498570-client-iot-graph-check.t - 8 second<br>
./tests/bugs/replicate/bug-1132102.t - 8 second<br>
./tests/bugs/quota/bug-1250582-volume-reset-should-not-remove-quota-quota-deem-statfs.t - 8 second<br>
./tests/bugs/quota/bug-1104692.t - 8 second<br>
./tests/bugs/posix/bug-1360679.t - 8 second<br>
./tests/bugs/posix/bug-1122028.t - 8 second<br>
./tests/bugs/nfs/bug-1157223-symlink-mounting.t - 8 second<br>
./tests/bugs/glusterfs/bug-861015-log.t - 8 second<br>
./tests/bugs/glusterd/sync-post-glusterd-restart.t - 8 second<br>
./tests/bugs/glusterd/bug-1696046.t - 8 second<br>
./tests/bugs/fuse/bug-983477.t - 8 second<br>
./tests/bugs/ec/bug-1227869.t - 8 second<br>
./tests/bugs/distribute/bug-1088231.t - 8 second<br>
./tests/bugs/distribute/bug-1086228.t - 8 second<br>
./tests/bugs/cli/bug-1087487.t - 8 second<br>
./tests/bugs/cli/bug-1022905.t - 8 second<br>
./tests/bugs/bug-1258069.t - 8 second<br>
./tests/bugs/bitrot/1209752-volume-status-should-show-bitrot-scrub-info.t - 8 second<br>
./tests/basic/xlator-pass-through-sanity.t - 8 second<br>
./tests/basic/quota-nfs.t - 8 second<br>
./tests/basic/glusterd/arbiter-volume.t - 8 second<br>
./tests/basic/ctime/ctime-noatime.t - 8 second<br>
./tests/line-coverage/cli-peer-and-volume-operations.t - 7 second<br>
./tests/gfid2path/get-gfid-to-path.t - 7 second<br>
./tests/bugs/upcall/bug-1369430.t - 7 second<br>
./tests/bugs/snapshot/bug-1260848.t - 7 second<br>
./tests/bugs/shard/shard-inode-refcount-test.t - 7 second<br>
./tests/bugs/shard/bug-1258334.t - 7 second<br>
./tests/bugs/replicate/bug-767585-gfid.t - 7 second<br>
./tests/bugs/replicate/bug-1448804-check-quorum-type-values.t - 7 second<br>
./tests/bugs/replicate/bug-1250170-fsync.t - 7 second<br>
./tests/bugs/posix/bug-1175711.t - 7 second<br>
./tests/bugs/nfs/bug-915280.t - 7 second<br>
./tests/bugs/md-cache/setxattr-prepoststat.t - 7 second<br>
./tests/bugs/md-cache/bug-1211863_unlink.t - 7 second<br>
./tests/bugs/glusterfs/bug-848251.t - 7 second<br>
./tests/bugs/distribute/bug-1122443.t - 7 second<br>
./tests/bugs/changelog/bug-1208470.t - 7 second<br>
./tests/bugs/bug-1702299.t - 7 second<br>
./tests/bugs/bug-1371806_2.t - 7 second<br>
./tests/bugs/bitrot/1209818-vol-info-show-scrub-process-properly.t - 7 second<br>
./tests/bugs/bitrot/1209751-bitrot-scrub-tunable-reset.t - 7 second<br>
./tests/bugs/bitrot/1207029-bitrot-daemon-should-start-on-valid-node.t - 7 second<br>
./tests/bitrot/br-stub.t - 7 second<br>
./tests/basic/glusterd/arbiter-volume-probe.t - 7 second<br>
./tests/basic/gfapi/libgfapi-fini-hang.t - 7 second<br>
./tests/basic/fencing/fencing-crash-conistency.t - 7 second<br>
./tests/basic/distribute/file-create.t - 7 second<br>
./tests/basic/afr/tarissue.t - 7 second<br>
./tests/basic/afr/gfid-heal.t - 7 second<br>
./tests/bugs/snapshot/bug-1178079.t - 6 second<br>
./tests/bugs/snapshot/bug-1064768.t - 6 second<br>
./tests/bugs/shard/bug-1342298.t - 6 second<br>
./tests/bugs/shard/bug-1259651.t - 6 second<br>
./tests/bugs/replicate/bug-1686568-send-truncate-on-arbiter-from-shd.t - 6 second<br>
./tests/bugs/replicate/bug-1626994-info-split-brain.t - 6 second<br>
./tests/bugs/replicate/bug-1325792.t - 6 second<br>
./tests/bugs/replicate/bug-1101647.t - 6 second<br>
./tests/bugs/quota/bug-1243798.t - 6 second<br>
./tests/bugs/protocol/bug-1321578.t - 6 second<br>
./tests/bugs/nfs/bug-877885.t - 6 second<br>
./tests/bugs/nfs/bug-1143880-fix-gNFSd-auth-crash.t - 6 second<br>
./tests/bugs/md-cache/bug-1476324.t - 6 second<br>
./tests/bugs/md-cache/afr-stale-read.t - 6 second<br>
./tests/bugs/io-cache/bug-858242.t - 6 second<br>
./tests/bugs/glusterfs/bug-893378.t - 6 second<br>
./tests/bugs/glusterfs/bug-856455.t - 6 second<br>
./tests/bugs/glusterd/quorum-value-check.t - 6 second<br>
./tests/bugs/ec/bug-1179050.t - 6 second<br>
./tests/bugs/distribute/bug-912564.t - 6 second<br>
./tests/bugs/distribute/bug-884597.t - 6 second<br>
./tests/bugs/distribute/bug-1368012.t - 6 second<br>
./tests/bugs/core/bug-986429.t - 6 second<br>
./tests/bugs/core/bug-1699025-brick-mux-detach-brick-fd-issue.t - 6 second<br>
./tests/bugs/core/bug-1168803-snapd-option-validation-fix.t - 6 second<br>
./tests/bugs/bug-1371806_1.t - 6 second<br>
./tests/bugs/bitrot/bug-1229134-bitd-not-support-vol-set.t - 6 second<br>
./tests/bugs/bitrot/bug-1210684-scrub-pause-resume-error-handling.t - 6 second<br>
./tests/bitrot/bug-1221914.t - 6 second<br>
./tests/basic/trace.t - 6 second<br>
./tests/basic/playground/template-xlator-sanity.t - 6 second<br>
./tests/basic/ec/nfs.t - 6 second<br>
./tests/basic/ec/ec-read-policy.t - 6 second<br>
./tests/basic/ec/ec-anonymous-fd.t - 6 second<br>
./tests/basic/distribute/non-root-unlink-stale-linkto.t - 6 second<br>
./tests/basic/changelog/changelog-rename.t - 6 second<br>
./tests/basic/afr/heal-info.t - 6 second<br>
./tests/basic/afr/afr-read-hash-mode.t - 6 second<br>
./tests/gfid2path/gfid2path_nfs.t - 5 second<br>
./tests/bugs/upcall/bug-1422776.t - 5 second<br>
./tests/bugs/replicate/bug-886998.t - 5 second<br>
./tests/bugs/replicate/bug-1365455.t - 5 second<br>
./tests/bugs/readdir-ahead/bug-1670253-consistent-metadata.t - 5 second<br>
./tests/bugs/posix/bug-gfid-path.t - 5 second<br>
./tests/bugs/posix/bug-765380.t - 5 second<br>
./tests/bugs/nfs/bug-847622.t - 5 second<br>
./tests/bugs/nfs/bug-1116503.t - 5 second<br>
./tests/bugs/io-stats/bug-1598548.t - 5 second<br>
./tests/bugs/glusterfs-server/bug-877992.t - 5 second<br>
./tests/bugs/glusterfs-server/bug-873549.t - 5 second<br>
./tests/bugs/glusterfs/bug-895235.t - 5 second<br>
./tests/bugs/fuse/bug-1126048.t - 5 second<br>
./tests/bugs/distribute/bug-907072.t - 5 second<br>
./tests/bugs/core/bug-913544.t - 5 second<br>
./tests/bugs/core/bug-908146.t - 5 second<br>
./tests/bugs/access-control/bug-1051896.t - 5 second<br>
./tests/basic/ec/ec-internal-xattrs.t - 5 second<br>
./tests/basic/ec/ec-fallocate.t - 5 second<br>
./tests/basic/distribute/bug-1265677-use-readdirp.t - 5 second<br>
./tests/basic/afr/arbiter-remove-brick.t - 5 second<br>
./tests/performance/quick-read.t - 4 second<br>
./tests/gfid2path/block-mount-access.t - 4 second<br>
./tests/features/delay-gen.t - 4 second<br>
./tests/bugs/upcall/bug-upcall-stat.t - 4 second<br>
./tests/bugs/upcall/bug-1394131.t - 4 second<br>
./tests/bugs/unclassified/bug-1034085.t - 4 second<br>
./tests/bugs/snapshot/bug-1111041.t - 4 second<br>
./tests/bugs/shard/bug-1272986.t - 4 second<br>
./tests/bugs/shard/bug-1256580.t - 4 second<br>
./tests/bugs/shard/bug-1250855.t - 4 second<br>
./tests/bugs/shard/bug-1245547.t - 4 second<br>
./tests/bugs/rpc/bug-954057.t - 4 second<br>
./tests/bugs/replicate/bug-976800.t - 4 second<br>
./tests/bugs/replicate/bug-880898.t - 4 second<br>
./tests/bugs/replicate/bug-1480525.t - 4 second<br>
./tests/bugs/read-only/bug-1134822-read-only-default-in-graph.t - 4 second<br>
./tests/bugs/readdir-ahead/bug-1446516.t - 4 second<br>
./tests/bugs/readdir-ahead/bug-1439640.t - 4 second<br>
./tests/bugs/readdir-ahead/bug-1390050.t - 4 second<br>
./tests/bugs/quota/bug-1287996.t - 4 second<br>
./tests/bugs/quick-read/bug-846240.t - 4 second<br>
./tests/bugs/posix/disallow-gfid-volumeid-removexattr.t - 4 second<br>
./tests/bugs/posix/bug-1619720.t - 4 second<br>
./tests/bugs/nl-cache/bug-1451588.t - 4 second<br>
./tests/bugs/nfs/zero-atime.t - 4 second<br>
./tests/bugs/nfs/subdir-trailing-slash.t - 4 second<br>
./tests/bugs/nfs/socket-as-fifo.t - 4 second<br>
./tests/bugs/nfs/showmount-many-clients.t - 4 second<br>
./tests/bugs/nfs/bug-1210338.t - 4 second<br>
./tests/bugs/nfs/bug-1166862.t - 4 second<br>
./tests/bugs/nfs/bug-1161092-nfs-acls.t - 4 second<br>
./tests/bugs/md-cache/bug-1632503.t - 4 second<br>
./tests/bugs/glusterfs-server/bug-864222.t - 4 second<br>
./tests/bugs/glusterfs/bug-1482528.t - 4 second<br>
./tests/bugs/glusterd/bug-948729/bug-948729-mode-script.t - 4 second<br>
./tests/bugs/glusterd/bug-948729/bug-948729-force.t - 4 second<br>
./tests/bugs/glusterd/bug-1482906-peer-file-blank-line.t - 4 second<br>
./tests/bugs/glusterd/bug-1091935-brick-order-check-from-cli-to-glusterd.t - 4 second<br>
./tests/bugs/geo-replication/bug-1296496.t - 4 second<br>
./tests/bugs/fuse/bug-1336818.t - 4 second<br>
./tests/bugs/fuse/bug-1283103.t - 4 second<br>
./tests/bugs/core/io-stats-1322825.t - 4 second<br>
./tests/bugs/core/bug-834465.t - 4 second<br>
./tests/bugs/core/bug-1135514-allow-setxattr-with-null-value.t - 4 second<br>
./tests/bugs/core/949327.t - 4 second<br>
./tests/bugs/cli/bug-977246.t - 4 second<br>
./tests/bugs/cli/bug-961307.t - 4 second<br>
./tests/bugs/cli/bug-1004218.t - 4 second<br>
./tests/bugs/bug-1138841.t - 4 second<br>
./tests/bugs/access-control/bug-1387241.t - 4 second<br>
./tests/bitrot/bug-internal-xattrs-check-1243391.t - 4 second<br>
./tests/basic/quota-rename.t - 4 second<br>
./tests/basic/hardlink-limit.t - 4 second<br>
./tests/basic/ec/dht-rename.t - 4 second<br>
./tests/basic/distribute/lookup.t - 4 second<br>
./tests/line-coverage/meta-max-coverage.t - 3 second<br>
./tests/gfid2path/gfid2path_fuse.t - 3 second<br>
./tests/bugs/unclassified/bug-991622.t - 3 second<br>
./tests/bugs/trace/bug-797171.t - 3 second<br>
./tests/bugs/glusterfs-server/bug-861542.t - 3 second<br>
./tests/bugs/glusterfs/bug-869724.t - 3 second<br>
./tests/bugs/glusterfs/bug-860297.t - 3 second<br>
./tests/bugs/glusterfs/bug-844688.t - 3 second<br>
./tests/bugs/glusterd/bug-948729/bug-948729.t - 3 second<br>
./tests/bugs/distribute/bug-1204140.t - 3 second<br>
./tests/bugs/core/bug-924075.t - 3 second<br>
./tests/bugs/core/bug-845213.t - 3 second<br>
./tests/bugs/core/bug-1421721-mpx-toggle.t - 3 second<br>
./tests/bugs/core/bug-1119582.t - 3 second<br>
./tests/bugs/core/bug-1117951.t - 3 second<br>
./tests/bugs/cli/bug-983317-volume-get.t - 3 second<br>
./tests/bugs/cli/bug-867252.t - 3 second<br>
./tests/basic/glusterd/check-cloudsync-ancestry.t - 3 second<br>
./tests/basic/fops-sanity.t - 3 second<br>
./tests/basic/fencing/test-fence-option.t - 3 second<br>
./tests/basic/distribute/debug-xattrs.t - 3 second<br>
./tests/basic/afr/ta-check-locks.t - 3 second<br>
./tests/line-coverage/volfile-with-all-graph-syntax.t - 2 second<br>
./tests/line-coverage/some-features-in-libglusterfs.t - 2 second<br>
./tests/bugs/shard/bug-1261773.t - 2 second<br>
./tests/bugs/replicate/bug-884328.t - 2 second<br>
./tests/bugs/readdir-ahead/bug-1512437.t - 2 second<br>
./tests/bugs/nfs/bug-970070.t - 2 second<br>
./tests/bugs/nfs/bug-1302948.t - 2 second<br>
./tests/bugs/logging/bug-823081.t - 2 second<br>
./tests/bugs/glusterfs-server/bug-889996.t - 2 second<br>
./tests/bugs/glusterfs/bug-892730.t - 2 second<br>
./tests/bugs/glusterfs/bug-811493.t - 2 second<br>
./tests/bugs/glusterd/bug-1085330-and-bug-916549.t - 2 second<br>
./tests/bugs/distribute/bug-924265.t - 2 second<br>
./tests/bugs/core/log-bug-1362520.t - 2 second<br>
./tests/bugs/core/bug-903336.t - 2 second<br>
./tests/bugs/core/bug-1111557.t - 2 second<br>
./tests/bugs/cli/bug-969193.t - 2 second<br>
./tests/bugs/cli/bug-949298.t - 2 second<br>
./tests/bugs/cli/bug-921215.t - 2 second<br>
./tests/bugs/cli/bug-1378842-volume-get-all.t - 2 second<br>
./tests/basic/peer-parsing.t - 2 second<br>
./tests/basic/md-cache/bug-1418249.t - 2 second<br>
./tests/basic/afr/arbiter-cli.t - 2 second<br>
./tests/bugs/replicate/ta-inode-refresh-read.t - 1 second<br>
./tests/bugs/glusterfs/bug-853690.t - 1 second<br>
./tests/bugs/cli/bug-764638.t - 1 second<br>
./tests/bugs/cli/bug-1047378.t - 1 second<br>
./tests/basic/netgroup_parsing.t - 1 second<br>
./tests/basic/gfapi/sink.t - 1 second<br>
./tests/basic/exports_parsing.t - 1 second<br>
./tests/basic/posixonly.t - 0 second<br>
./tests/basic/glusterfsd-args.t - 0 second<br>
<br>
<br>
2 test(s) failed <br>
./tests/basic/uss.t<br>
./tests/features/subdir-mount.t<br>
<br>
0 test(s) generated core <br>
<br>
<br>
5 test(s) needed retry <br>
./tests/basic/afr/split-brain-favorite-child-policy.t<br>
./tests/basic/ec/self-heal.t<br>
./tests/basic/uss.t<br>
./tests/basic/volfile-sanity.t<br>
./tests/features/subdir-mount.t<br>
<br>
Result is 1<br>
<br>
tar: Removing leading `/' from member names<br>
kernel.core_pattern = /%e-%p.core<br>
Build step 'Execute shell' marked build as failure<br>
_______________________________________________<br>
maintainers mailing list<br>
<a href="mailto:maintainers@gluster.org" target="_blank">maintainers@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/maintainers</a><br>
<br>
<br>
</div></div>-- <br><div dir="ltr" class="gmail-m_-5684315802421270440gmail-m_-902116316683095880gmail_signature">- Atin (atinm)</div>
_______________________________________________<br>
maintainers mailing list<br>
<a href="mailto:maintainers@gluster.org" target="_blank">maintainers@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/maintainers</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-5684315802421270440gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</blockquote></div></div>