<div><div dir="auto">This is the first nightly job failure since we reopened the master branch. Crash seems to be from fini () code path. Need investigation and RCA here.</div></div><div><br><div class="gmail_quote"><div dir="ltr">---------- Forwarded message ---------<br>From: <span dir="ltr"><<a href="mailto:jenkins@build.gluster.org">jenkins@build.gluster.org</a>></span><br>Date: Fri, 17 Aug 2018 at 23:54<br>Subject: [Gluster-Maintainers] Build failed in Jenkins: regression-test-with-multiplex #831<br>To: <<a href="mailto:maintainers@gluster.org">maintainers@gluster.org</a>>, <<a href="mailto:amarts@redhat.com">amarts@redhat.com</a>><br></div><br><br>See <<a href="https://build.gluster.org/job/regression-test-with-multiplex/831/display/redirect?page=changes" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/831/display/redirect?page=changes</a>><br>
<br>
Changes:<br>
<br>
[Amar Tumballi] jbr : fix coverity issues in jbr<br>
<br>
[Amar Tumballi] statedump : fix coverity issues<br>
<br>
[Amar Tumballi] glusterd: coverity defects fix introduced by commit 1f3bfe7<br>
<br>
[Amar Tumballi] features/acl: Fix a possible null dereference<br>
<br>
[Amar Tumballi] meta : fix coverity in meta-helpers.c<br>
<br>
[Amar Tumballi] nfs-server-mount : fix coverity issues in mount3.c<br>
<br>
[Amar Tumballi] posix: FORWARD_NULL coverity fix<br>
<br>
[Amar Tumballi] locks: FORWARD_NULL coverity fix<br>
<br>
[Amar Tumballi] doc: Add details around xlator categories<br>
<br>
[Amar Tumballi] features/changelog: Fix missing unlocks<br>
<br>
------------------------------------------<br>
[...truncated 1.01 MB...]<br>
top = 0x0<br>
victim = 0x0<br>
trav_p = 0x0<br>
count = 0<br>
victim_found = false<br>
ctx = 0x16cc010<br>
__FUNCTION__ = "posix_health_check_thread_proc"<br>
#3 0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#4 0x00007fba8d753bad in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 8 (Thread 0x7fba2b5f0700 (LWP 31062)):<br>
#0 0x00007fba8d71a56d in nanosleep () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#1 0x00007fba8d71a404 in sleep () from /lib64/libc.so.6<br>
No symbol table info available.<br>
#2 0x00007fba816f2081 in posix_disk_space_check_thread_proc (data=0x7fba505beff0) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/storage/posix/src/posix-helpers.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/storage/posix/src/posix-helpers.c</a>>:2286<br>
this = 0x7fba505beff0<br>
priv = 0x7fba53fe9490<br>
interval = 5<br>
ret = 0<br>
__FUNCTION__ = "posix_disk_space_check_thread_proc"<br>
#3 0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#4 0x00007fba8d753bad in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 7 (Thread 0x7fba703b3700 (LWP 31055)):<br>
#0 0x00007fba8e092d42 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007fba7b345737 in iot_worker (data=0x7fba53ba86b0) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/performance/io-threads/src/io-threads.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/performance/io-threads/src/io-threads.c</a>>:195<br>
conf = 0x7fba53ba86b0<br>
this = 0x7fba53277620<br>
stub = 0x0<br>
sleep_till = {tv_sec = 1534518836, tv_nsec = 345143975}<br>
ret = 0<br>
pri = -1<br>
bye = false<br>
__FUNCTION__ = "iot_worker"<br>
#2 0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#3 0x00007fba8d753bad in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 6 (Thread 0x7fba704f5700 (LWP 31054)):<br>
#0 0x00007fba8e092995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007fba7aae8308 in index_worker (data=0x7fba5327d8a0) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/features/index/src/index.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/features/index/src/index.c</a>>:218<br>
priv = 0x7fba53b980f0<br>
this = 0x7fba5327d8a0<br>
stub = 0x0<br>
bye = false<br>
#2 0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#3 0x00007fba8d753bad in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 5 (Thread 0x7fba1d4d5700 (LWP 30825)):<br>
#0 0x00007fba8e092995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007fba8edf9da0 in rpcsvc_request_handler (arg=0x7fba7c046210) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c</a>>:1983<br>
program = 0x7fba7c046210<br>
req = 0x0<br>
actor = 0x0<br>
done = false<br>
ret = 0<br>
__FUNCTION__ = "rpcsvc_request_handler"<br>
#2 0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#3 0x00007fba8d753bad in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 4 (Thread 0x7fba1dcd6700 (LWP 30824)):<br>
#0 0x00007fba8e092995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007fba8edf9da0 in rpcsvc_request_handler (arg=0x7fba7c0465d0) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c</a>>:1983<br>
program = 0x7fba7c0465d0<br>
req = 0x0<br>
actor = 0x7fba7a449700 <glusterfs4_0_fop_actors+1792><br>
done = false<br>
ret = 0<br>
__FUNCTION__ = "rpcsvc_request_handler"<br>
#2 0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#3 0x00007fba8d753bad in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 3 (Thread 0x7fba1f4d9700 (LWP 30313)):<br>
#0 0x00007fba8e092995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007fba8edf9da0 in rpcsvc_request_handler (arg=0x7fba7c0465d0) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c</a>>:1983<br>
program = 0x7fba7c0465d0<br>
req = 0x0<br>
actor = 0x7fba7a449700 <glusterfs4_0_fop_actors+1792><br>
done = false<br>
ret = 0<br>
__FUNCTION__ = "rpcsvc_request_handler"<br>
#2 0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#3 0x00007fba8d753bad in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 2 (Thread 0x7fba795b7700 (LWP 29638)):<br>
#0 0x00007fba8e092995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#1 0x00007fba8edf9da0 in rpcsvc_request_handler (arg=0x7fba7c0465d0) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c</a>>:1983<br>
program = 0x7fba7c0465d0<br>
req = 0x0<br>
actor = 0x7fba7a449a80 <glusterfs4_0_fop_actors+2688><br>
done = false<br>
ret = 0<br>
__FUNCTION__ = "rpcsvc_request_handler"<br>
#2 0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#3 0x00007fba8d753bad in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
<br>
Thread 1 (Thread 0x7fba82147700 (LWP 29636)):<br>
#0 0x00007fba7a67bf95 in ios_stat_unref (iosstat=0xffffffffffffffff) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c</a>>:350<br>
cleanup = 0<br>
refcnt = 0<br>
#1 0x00007fba7a69f766 in ios_destroy_top_stats (conf=0x7fba5005fff0) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c</a>>:3717<br>
i = 4<br>
list_head = 0x7fba50061848<br>
entry = 0x7fba50486400<br>
tmp = 0xffffffffffffff00<br>
list = 0x7fba50486400<br>
stat = 0xffffffffffffffff<br>
__PRETTY_FUNCTION__ = "ios_destroy_top_stats"<br>
#2 0x00007fba7a6a0504 in ios_conf_destroy (conf=0x7fba5005fff0) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c</a>>:3966<br>
No locals.<br>
#3 0x00007fba7a6a0ee4 in fini (this=0x7fba5005cf30) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/debug/io-stats/src/io-stats.c</a>>:4145<br>
conf = 0x7fba5005fff0<br>
__FUNCTION__ = "fini"<br>
#4 0x00007fba8f03822d in xlator_call_fini (this=0x7fba5005cf30) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/xlator.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/xlator.c</a>>:1015<br>
No locals.<br>
#5 0x00007fba8f0382ce in xlator_mem_cleanup (this=0x7fba5005df70) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/xlator.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/xlator.c</a>>:1036<br>
list = 0x7fba50486780<br>
trav = 0x7fba5005cf30<br>
inode_table = 0x0<br>
prev = 0x7fba5005cf30<br>
ctx = 0x16cc010<br>
trav_p = 0x0<br>
top = 0x0<br>
victim = 0x0<br>
#6 0x00007fba7a1cd5f9 in server_rpc_notify (rpc=0x7fba7c044540, xl=0x7fba7c02f940, event=RPCSVC_EVENT_TRANSPORT_DESTROY, data=0x7fba64009380) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/protocol/server/src/server.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/xlators/protocol/server/src/server.c</a>>:565<br>
detached = false<br>
this = 0x7fba7c02f940<br>
trans = 0x7fba64009380<br>
conf = 0x7fba7c037020<br>
client = 0x7fba6438d8a0<br>
auth_path = 0x0<br>
ret = -1<br>
victim_found = true<br>
xlator_name = 0x7fba7c017170 "/d/backends/patchy2"<br>
ctx = 0x16cc010<br>
top = 0x7fba7c02f940<br>
trav_p = 0x7fba7c02f970<br>
travxl = 0x7fba5005df70<br>
xprtrefcount = 0<br>
tmp = 0x7fba50486cd0<br>
__FUNCTION__ = "server_rpc_notify"<br>
#7 0x00007fba8edf6a59 in rpcsvc_program_notify (listener=0x7fba7c046110, event=RPCSVC_EVENT_TRANSPORT_DESTROY, data=0x7fba64009380) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c</a>>:345<br>
wrapper = 0x7fba7c045360<br>
#8 0x00007fba8edf7c3f in rpcsvc_notify (trans=0x7fba64009380, mydata=0x7fba7c044540, event=RPC_TRANSPORT_CLEANUP, data=0x0) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpcsvc.c</a>>:838<br>
ret = -1<br>
msg = 0x0<br>
new_trans = 0x0<br>
svc = 0x7fba7c044540<br>
listener = 0x7fba7c046110<br>
__FUNCTION__ = "rpcsvc_notify"<br>
#9 0x00007fba8edfd6e8 in rpc_transport_unref (this=0x7fba64009380) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpc-transport.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-lib/src/rpc-transport.c</a>>:526<br>
refcount = 0<br>
ret = -1<br>
__FUNCTION__ = "rpc_transport_unref"<br>
#10 0x00007fba83c104cd in socket_event_handler (fd=39, idx=18, gen=10, data=0x7fba64009380, poll_in=1, poll_out=0, poll_err=16) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-transport/socket/src/socket.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/rpc/rpc-transport/socket/src/socket.c</a>>:3026<br>
sa = 0x7fba64009440<br>
this = 0x7fba64009380<br>
priv = 0x7fba64007880<br>
ret = -1<br>
ctx = 0x16cc010<br>
socket_closed = true<br>
notify_handled = false<br>
__FUNCTION__ = "socket_event_handler"<br>
#11 0x00007fba8f0b5bff in event_dispatch_epoll_handler (event_pool=0x1703c30, event=0x7fba82146ea0) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/event-epoll.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/event-epoll.c</a>>:587<br>
ev_data = 0x7fba82146ea4<br>
slot = 0x1735510<br>
handler = 0x7fba83c0feba <socket_event_handler><br>
data = 0x7fba64009380<br>
idx = 18<br>
gen = 10<br>
ret = -1<br>
fd = 39<br>
handled_error_previously = false<br>
__FUNCTION__ = "event_dispatch_epoll_handler"<br>
#12 0x00007fba8f0b5ef2 in event_dispatch_epoll_worker (data=0x174f120) at <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/event-epoll.c" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/libglusterfs/src/event-epoll.c</a>>:663<br>
event = {events = 17, data = {ptr = 0xa00000012, fd = 18, u32 = 18, u64 = 42949672978}}<br>
ret = 1<br>
ev_data = 0x174f120<br>
event_pool = 0x1703c30<br>
myindex = 1<br>
timetodie = 0<br>
__FUNCTION__ = "event_dispatch_epoll_worker"<br>
#13 0x00007fba8e08ee25 in start_thread () from /lib64/libpthread.so.0<br>
No symbol table info available.<br>
#14 0x00007fba8d753bad in clone () from /lib64/libc.so.6<br>
No symbol table info available.<br>
=========================================================<br>
Finish backtrace<br>
program name : /build/install/sbin/glusterfsd<br>
corefile : /glfs_epoll000-29630.core<br>
=========================================================<br>
<br>
+ rm -f /build/install/cores/gdbout.txt<br>
+ sort /build/install/cores/liblist.txt<br>
+ uniq<br>
+ cat /build/install/cores/liblist.txt.tmp<br>
+ grep -v /build/install<br>
+ tar -cf /archives/archived_builds/build-install-regression-test-with-multiplex-831.tar /build/install/sbin /build/install/bin /build/install/lib /build/install/libexec /build/install/cores<br>
tar: Removing leading `/' from member names<br>
+ tar -rhf /archives/archived_builds/build-install-regression-test-with-multiplex-831.tar -T /build/install/cores/liblist.txt<br>
tar: Removing leading `/' from member names<br>
+ bzip2 /archives/archived_builds/build-install-regression-test-with-multiplex-831.tar<br>
+ rm -f /build/install/cores/liblist.txt<br>
+ rm -f /build/install/cores/liblist.txt.tmp<br>
+ find /archives -size +1G -delete -type f<br>
+ echo 'Cores and build archived in <a href="http://builder102.cloud.gluster.org/archived_builds/build-install-regression-test-with-multiplex-831.tar.bz2" rel="noreferrer" target="_blank">http://builder102.cloud.gluster.org/archived_builds/build-install-regression-test-with-multiplex-831.tar.bz2</a>'<br>
Cores and build archived in <a href="http://builder102.cloud.gluster.org/archived_builds/build-install-regression-test-with-multiplex-831.tar.bz2" rel="noreferrer" target="_blank">http://builder102.cloud.gluster.org/archived_builds/build-install-regression-test-with-multiplex-831.tar.bz2</a><br>
+ echo 'Open core using the following command to get a proper stack'<br>
Open core using the following command to get a proper stack<br>
+ echo 'Example: From root of extracted tarball'<br>
Example: From root of extracted tarball<br>
+ echo '\t\tgdb -ex '\''set sysroot ./'\'' -ex '\''core-file ./build/install/cores/xxx.core'\'' <target, say ./build/install/sbin/glusterd>'<br>
\t\tgdb -ex 'set sysroot ./' -ex 'core-file ./build/install/cores/xxx.core' <target, say ./build/install/sbin/glusterd><br>
+ RET=1<br>
+ '[' 1 -ne 0 ']'<br>
+ tar -czf <<a href="https://build.gluster.org/job/regression-test-with-multiplex/831/artifact/glusterfs-logs.tgz" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/831/artifact/glusterfs-logs.tgz</a>> /var/log/glusterfs /var/log/messages /var/log/messages-20180722 /var/log/messages-20180729 /var/log/messages-20180805 /var/log/messages-20180812<br>
tar: Removing leading `/' from member names<br>
+ scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i <<a href="https://build.gluster.org/job/regression-test-with-multiplex/ws/" rel="noreferrer" target="_blank">https://build.gluster.org/job/regression-test-with-multiplex/ws/</a>> glusterfs-logs.tgz _logs_collector@http.int.rht.gluster.org:/var/www/glusterfs-logs/regression-test-with-multiplex-831.tgz<br>
ssh: connect to host <a href="http://http.int.rht.gluster.org" rel="noreferrer" target="_blank">http.int.rht.gluster.org</a> port 22: Connection timed out<br>
lost connection<br>
+ true<br>
+ case $(uname -s) in<br>
++ uname -s<br>
+ /sbin/sysctl -w kernel.core_pattern=/%e-%p.core<br>
kernel.core_pattern = /%e-%p.core<br>
+ exit 1<br>
Build step 'Execute shell' marked build as failure<br>
_______________________________________________<br>
maintainers mailing list<br>
<a href="mailto:maintainers@gluster.org" target="_blank">maintainers@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/maintainers</a><br>
</div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">- Atin (atinm)</div>