<div dir="ltr">Hi,<div><br></div><div>It seems two NFS test cases are crashed, Can anyone look it from NFS-team?</div><div><br></div><div>Thanks,</div><div>Mohit Agrawal </div><div><br></div></div><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jun 26, 2020 at 1:04 AM <<a href="mailto:jenkins@build.gluster.org" target="_blank">jenkins@build.gluster.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">See <<a href="https://build.gluster.org/job/centos8-regression/33/display/redirect?page=changes" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/33/display/redirect?page=changes</a>><br>
<br>
Changes:<br>
<br>
[susant] dht - fixing xattr inconsistency<br>
<br>
<br>
------------------------------------------<br>
[...truncated 3.91 MB...]<br>
#2 0x000000000040b2a5 in ?? ()<br>
No symbol table info available.<br>
#3 0x0000000000000000 in ?? ()<br>
No symbol table info available.<br>
<br>
Thread 7 (Thread 0x7efe51847700 (LWP 24835)):<br>
#0 0x00007efe5ad77548 in nanosleep () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x00007efe5ad7744e in sleep () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#2 0x00007efe5ca1da80 in pool_sweeper (arg=0x0) at <<a href="https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/mem-pool.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/mem-pool.c</a>>:453<br>
state = {cold_lists = {0x0 <repeats 1024 times>}, n_cold_lists = 0}<br>
pool_list = 0x0<br>
i = 0<br>
pending = true<br>
#3 0x00007efe5b5582de in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#4 0x00007efe5adab133 in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 6 (Thread 0x7efe51046700 (LWP 24836)):<br>
#0 0x00007efe5b55e7ca in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007efe5ca3629c in syncenv_task (proc=0xd4bc70) at <<a href="https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/syncop.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/syncop.c</a>>:577<br>
env = 0xd4bc70<br>
task = 0x0<br>
sleep_till = {tv_sec = 1593109059, tv_nsec = 0}<br>
ret = 0<br>
#2 0x00007efe5ca365cf in syncenv_processor (thdata=0xd4bc70) at <<a href="https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/syncop.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/syncop.c</a>>:678<br>
proc = 0xd4bc70<br>
task = 0x7efe400043b0<br>
#3 0x00007efe5b5582de in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#4 0x00007efe5adab133 in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 5 (Thread 0x7efe50845700 (LWP 24837)):<br>
#0 0x00007efe5b55e7ca in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007efe5ca3629c in syncenv_task (proc=0xd4c050) at <<a href="https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/syncop.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/syncop.c</a>>:577<br>
env = 0xd4bc70<br>
task = 0x0<br>
sleep_till = {tv_sec = 1593109059, tv_nsec = 0}<br>
ret = 0<br>
#2 0x00007efe5ca365cf in syncenv_processor (thdata=0xd4c050) at <<a href="https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/syncop.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/syncop.c</a>>:678<br>
proc = 0xd4c050<br>
task = 0x0<br>
#3 0x00007efe5b5582de in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#4 0x00007efe5adab133 in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 4 (Thread 0x7efe50044700 (LWP 24838)):<br>
#0 0x00007efe5ada279f in select () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x00007efe5ca7fcc4 in runner (arg=0xd4fb20) at <<a href="https://build.gluster.org/job/centos8-regression/ws/contrib/timer-wheel/timer-wheel.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/ws/contrib/timer-wheel/timer-wheel.c</a>>:186<br>
tv = {tv_sec = 0, tv_usec = 785040}<br>
base = 0xd4fb20<br>
#2 0x00007efe5b5582de in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#3 0x00007efe5adab133 in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 3 (Thread 0x7efe4f179700 (LWP 24839)):<br>
#0 0x00007efe5adab467 in epoll_wait () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x00007efe5ca61368 in event_dispatch_epoll_worker (data=0xdb6920) at <<a href="https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/event-epoll.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/event-epoll.c</a>>:741<br>
event = {events = 1, data = {ptr = 0x100000009, fd = 9, u32 = 9, u64 = 4294967305}}<br>
ret = 0<br>
ev_data = 0xdb6920<br>
event_pool = 0xd3ef50<br>
myindex = 1<br>
timetodie = 0<br>
gen = 0<br>
poller_death_notify = {next = 0x0, prev = 0x0}<br>
slot = 0x0<br>
tmp = 0x0<br>
__FUNCTION__ = "event_dispatch_epoll_worker"<br>
#2 0x00007efe5b5582de in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#3 0x00007efe5adab133 in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 2 (Thread 0x7efe5cf115c0 (LWP 24832)):<br>
#0 0x00007efe5b5597bd in __pthread_timedjoin_ex () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007efe5ca6162b in event_dispatch_epoll (event_pool=0xd3ef50) at <<a href="https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/event-epoll.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/event-epoll.c</a>>:835<br>
i = 2<br>
t_id = 139630705346304<br>
pollercount = 2<br>
ret = 0<br>
ev_data = 0xdb6b10<br>
__FUNCTION__ = "event_dispatch_epoll"<br>
#2 0x00007efe5ca1c473 in gf_event_dispatch (event_pool=0xd3ef50) at <<a href="https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/event.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/ws/libglusterfs/src/event.c</a>>:115<br>
ret = -1<br>
__FUNCTION__ = "gf_event_dispatch"<br>
#3 0x000000000040c20e in ?? ()<br>
No symbol table info available.<br>
#4 0x00007ffc81ed9338 in ?? ()<br>
No symbol table info available.<br>
#5 0x0000000b000081ed in ?? ()<br>
No symbol table info available.<br>
#6 0x692f646c6975622f in ?? ()<br>
No symbol table info available.<br>
#7 0x732f6c6c6174736e in ?? ()<br>
No symbol table info available.<br>
#8 0x73756c672f6e6962 in ?? ()<br>
No symbol table info available.<br>
#9 0x732d207366726574 in ?? ()<br>
No symbol table info available.<br>
#10 0x6f686c61636f6c20 in ?? ()<br>
No symbol table info available.<br>
#11 0x6c6f762d2d207473 in ?? ()<br>
No symbol table info available.<br>
#12 0x2064692d656c6966 in ?? ()<br>
No symbol table info available.<br>
#13 0x2f72657473756c67 in ?? ()<br>
No symbol table info available.<br>
#14 0x2f20702d2073666e in ?? ()<br>
No symbol table info available.<br>
#15 0x2f6e75722f726176 in ?? ()<br>
No symbol table info available.<br>
#16 0x2f72657473756c67 in ?? ()<br>
No symbol table info available.<br>
#17 0x2e73666e2f73666e in ?? ()<br>
No symbol table info available.<br>
#18 0x2f206c2d20646970 in ?? ()<br>
No symbol table info available.<br>
#19 0x2f676f6c2f726176 in ?? ()<br>
No symbol table info available.<br>
#20 0x6672657473756c67 in ?? ()<br>
No symbol table info available.<br>
#21 0x6f6c2e73666e2f73 in ?? ()<br>
No symbol table info available.<br>
#22 0x61762f20532d2067 in ?? ()<br>
No symbol table info available.<br>
#23 0x6c672f6e75722f72 in ?? ()<br>
No symbol table info available.<br>
#24 0x39302f7265747375 in ?? ()<br>
No symbol table info available.<br>
#25 0x3133613934656634 in ?? ()<br>
No symbol table info available.<br>
#26 0x732e343439616239 in ?? ()<br>
No symbol table info available.<br>
#27 0x00000074656b636f in ?? ()<br>
No symbol table info available.<br>
#28 0x0000000000000000 in ?? ()<br>
No symbol table info available.<br>
<br>
Thread 1 (Thread 0x7efe4cb27700 (LWP 24843)):<br>
#0 0x00007efe5ace68df in raise () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x00007efe5acd0cf5 in abort () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#2 0x00007efe5acd0bc9 in __assert_fail_base.cold.0 () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#3 0x00007efe5acdee96 in __assert_fail () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#4 0x00007efe4d2bdd9d in mountudp_program_3 (rqstp=0x7efe4cb25a40, transp=0x7efe3c0039a0) at <<a href="https://build.gluster.org/job/centos8-regression/ws/xlators/nfs/server/src/mount3udp_svc.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/ws/xlators/nfs/server/src/mount3udp_svc.c</a>>:146<br>
argument = {mountudpproc3_mnt_3_arg = 0x2 <error: Cannot access memory at address 0x2>}<br>
result = 0x0<br>
_xdr_argument = 0x0<br>
_xdr_result = 0x0<br>
local = 0x0<br>
res = 0x0<br>
sin = 0x7efe3c0039b4<br>
__PRETTY_FUNCTION__ = "mountudp_program_3"<br>
__FUNCTION__ = "mountudp_program_3"<br>
#5 0x00007efe5c335e09 in svc_getreq_common () from /lib64/libtirpc.so.3<br>
No symbol table info available.<br>
#6 0x00007efe5c335fa7 in svc_getreq_poll () from /lib64/libtirpc.so.3<br>
No symbol table info available.<br>
#7 0x00007efe5c3387ee in svc_run () from /lib64/libtirpc.so.3<br>
No symbol table info available.<br>
#8 0x00007efe4d2be14a in mount3udp_thread (argv=0x7efe48014390) at <<a href="https://build.gluster.org/job/centos8-regression/ws/xlators/nfs/server/src/mount3udp_svc.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/ws/xlators/nfs/server/src/mount3udp_svc.c</a>>:234<br>
nfsx = 0x7efe48014390<br>
transp = 0x7efe3c0039a0<br>
__PRETTY_FUNCTION__ = "mount3udp_thread"<br>
__FUNCTION__ = "mount3udp_thread"<br>
#9 0x00007efe5b5582de in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#10 0x00007efe5adab133 in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
=========================================================<br>
Finish backtrace<br>
program name : /build/install/sbin/glusterfs<br>
corefile : /glfs_nfsudp-24832.core<br>
=========================================================<br>
<br>
+ rm -f /build/install/cores/gdbout.txt<br>
+ for corefile in $CORELIST<br>
+ getliblistfromcore /build/install/cores/glfs_nfsnsm-21720.core<br>
+ rm -f /build/install/cores/gdbout.txt<br>
+ gdb -c /build/install/cores/glfs_nfsnsm-21720.core -q -ex 'set pagination off' -ex 'info sharedlibrary' -ex q<br>
+ set +x<br>
+ rm -f /build/install/cores/gdbout.txt<br>
+ for corefile in $CORELIST<br>
+ getliblistfromcore /build/install/cores/glfs_nfsnsm-23886.core<br>
+ rm -f /build/install/cores/gdbout.txt<br>
+ gdb -c /build/install/cores/glfs_nfsnsm-23886.core -q -ex 'set pagination off' -ex 'info sharedlibrary' -ex q<br>
+ set +x<br>
+ rm -f /build/install/cores/gdbout.txt<br>
+ for corefile in $CORELIST<br>
+ getliblistfromcore /build/install/cores/glfs_nfsudp-24832.core<br>
+ rm -f /build/install/cores/gdbout.txt<br>
+ gdb -c /build/install/cores/glfs_nfsudp-24832.core -q -ex 'set pagination off' -ex 'info sharedlibrary' -ex q<br>
+ set +x<br>
+ rm -f /build/install/cores/gdbout.txt<br>
+ sort /build/install/cores/liblist.txt<br>
+ uniq<br>
+ cat /build/install/cores/liblist.txt.tmp<br>
+ grep -v /build/install<br>
+ tar -cf /archives/archived_builds/build-install-centos8-regression-33.tar /build/install/sbin /build/install/bin /build/install/lib /build/install/libexec /build/install/cores<br>
tar: Removing leading `/' from member names<br>
tar: Removing leading `/' from hard link targets<br>
+ tar -rhf /archives/archived_builds/build-install-centos8-regression-33.tar -T /build/install/cores/liblist.txt<br>
tar: Removing leading `/' from member names<br>
tar: Removing leading `/' from hard link targets<br>
+ bzip2 /archives/archived_builds/build-install-centos8-regression-33.tar<br>
/opt/qa/regression.sh: line 159: bzip2: command not found<br>
+ rm -f /build/install/cores/liblist.txt<br>
+ rm -f /build/install/cores/liblist.txt.tmp<br>
+ find /archives -size +1G -delete -type f<br>
+ [[ <a href="http://builder212.int.aws.gluster.org" rel="noreferrer" target="_blank">builder212.int.aws.gluster.org</a> == *\a\w\s* ]]<br>
+ scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i **** /archives/archived_builds/build-install-centos8-regression-33.tar.bz2 _logs-collector@logs.aws.gluster.org:/var/www/glusterfs-logs/centos8-regression-33.bz2<br>
Warning: Permanently added '<a href="http://logs.aws.gluster.org" rel="noreferrer" target="_blank">logs.aws.gluster.org</a>,18.219.45.211' (ECDSA) to the list of known hosts.<br>
/archives/archived_builds/build-install-centos8-regression-33.tar.bz2: No such file or directory<br>
+ true<br>
+ echo 'Cores and builds archived in <a href="https://logs.aws.gluster.org/centos8-regression-33.bz2" rel="noreferrer" target="_blank">https://logs.aws.gluster.org/centos8-regression-33.bz2</a>'<br>
Cores and builds archived in <a href="https://logs.aws.gluster.org/centos8-regression-33.bz2" rel="noreferrer" target="_blank">https://logs.aws.gluster.org/centos8-regression-33.bz2</a><br>
+ echo 'Open core using the following command to get a proper stack'<br>
Open core using the following command to get a proper stack<br>
+ echo 'Example: From root of extracted tarball'<br>
Example: From root of extracted tarball<br>
+ echo '\t\tgdb -ex '\''set sysroot ./'\'' -ex '\''core-file ./build/install/cores/xxx.core'\'' <target, say ./build/install/sbin/glusterd>'<br>
\t\tgdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/xxx.core' <target, say ./build/install/sbin/glusterd><br>
+ RET=1<br>
+ '[' 1 -ne 0 ']'<br>
+ tar -czf <<a href="https://build.gluster.org/job/centos8-regression/33/artifact/glusterfs-logs.tgz" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/33/artifact/glusterfs-logs.tgz</a>> /var/log/glusterfs /var/log/messages<br>
tar: Removing leading `/' from member names<br>
tar: Removing leading `/' from hard link targets<br>
+ case $(uname -s) in<br>
++ uname -s<br>
+ /sbin/sysctl -w kernel.core_pattern=/%e-%p.core<br>
kernel.core_pattern = /%e-%p.core<br>
+ exit 1<br>
Logs are archived at Build artifacts: <a href="https://build.gluster.org/job/centos8-regression/" rel="noreferrer" target="_blank">https://build.gluster.org/job/centos8-regression/</a><br>
<a href="mailto:build@review.gluster.org" target="_blank">build@review.gluster.org</a>: Permission denied (publickey).<br>
Build step 'Execute shell' marked build as failure<br>
<br>
</blockquote></div>