<div dir="ltr">This is failing too frequently. Latest failure at <a href="https://build.gluster.org/job/centos6-regression/4974/console" target="_blank">https://build.gluster.org/job/<wbr>centos6-regression/4974/<wbr>console</a> . I guess we don't have any maintainers for crypt xlator, so request Vijay/Shyam/Jeff to take a look? Otherwise we need to mark this test bad to avoid spurious failures.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 6, 2017 at 12:07 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>fstat report: <a href="https://fstat.gluster.org/summary?start_date=2017-05-25&end_date=2017-06-06" target="_blank">https://fstat.gluster.org/<wbr>summary?start_date=2017-05-25&<wbr>end_date=2017-06-06</a><br><br>One of the regression regression link : <a href="https://build.gluster.org/job/centos6-regression/4908/console" target="_blank">https://build.gluster.org/job/<wbr>centos6-regression/4908/<wbr>console</a><br><br></div>Who can help here?<br><div><div><br><br>(gdb) t a a bt<br><br>Thread 11 (LWP 25166):<br>#0 0x00007f1ead84500d in nanosleep () from ./lib64/libpthread.so.0<br>#1 0x00007f1eae576365 in gf_timer_proc (data=0xe34ea0)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/<wbr>libglusterfs/src/timer.c:164<br>#2 0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#3 0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 10 (LWP 25169):<br>#0 0x00007f1ead841a5e in pthread_cond_timedwait@@GLIBC_<wbr>2.3.2 () from ./lib64/libpthread.so.0<br>#1 0x00007f1eae5b0f9c in syncenv_task (proc=0xe35690)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/<wbr>libglusterfs/src/syncop.c:603<br>#2 0x00007f1eae5b123e in syncenv_processor (thdata=0xe35690)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/<wbr>libglusterfs/src/syncop.c:695<br>#3 0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#4 0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 9 (LWP 25173):<br>#0 0x00007f1ead841a5e in pthread_cond_timedwait@@GLIBC_<wbr>2.3.2 () from ./lib64/libpthread.so.0<br>#1 0x00007f1e9b9c5d7a in iot_worker (data=0x7f1e9c024ec0)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>performance/io-threads/src/io-<wbr>threads.c:193<br>#2 0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#3 0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 8 (LWP 25170):<br>#0 0x00007f1ead841a5e in pthread_cond_timedwait@@GLIBC_<wbr>2.3.2 () from ./lib64/libpthread.so.0<br>#1 0x00007f1eae5b0f9c in syncenv_task (proc=0xe35a50)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/<wbr>libglusterfs/src/syncop.c:603<br>#2 0x00007f1eae5b123e in syncenv_processor (thdata=0xe35a50)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/<wbr>libglusterfs/src/syncop.c:695<br>#3 0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#4 0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 7 (LWP 25168):<br>#0 0x00007f1ead169c4d in nanosleep () from ./lib64/libc.so.6<br>#1 0x00007f1ead169ac0 in sleep () from ./lib64/libc.so.6<br>#2 0x00007f1eae59b946 in pool_sweeper (arg=0x0)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/<wbr>libglusterfs/src/mem-pool.c:<wbr>465<br>#3 0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#4 0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 6 (LWP 25167):<br>#0 0x00007f1ead845585 in sigwait () from ./lib64/libpthread.so.0<br>#1 0x0000000000409f22 in glusterfs_sigwaiter (arg=0x7ffc45a51640)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/glusterfsd/<wbr>src/glusterfsd.c:2111<br>#2 0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#3 0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 5 (LWP 25165):<br>#0 0x00007f1ead83e2fd in pthread_join () from ./lib64/libpthread.so.0<br>#1 0x00007f1eae5d706f in event_dispatch_epoll (event_pool=0xe18050)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/<wbr>libglusterfs/src/event-epoll.<wbr>c:732<br>#2 0x00007f1eae59a096 in event_dispatch (event_pool=0xe18050)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/<wbr>libglusterfs/src/event.c:124<br>#3 0x000000000040ac33 in main (argc=6, argv=0x7ffc45a528b8)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/glusterfsd/<wbr>src/glusterfsd.c:2522<br><br>Thread 4 (LWP 25177):<br>#0 0x00007f1ead84168c in pthread_cond_wait@@GLIBC_2.3.2 () from ./lib64/libpthread.so.0<br>#1 0x00007f1ea63110b6 in notify_kernel_loop (data=0xe1f910)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>mount/fuse/src/fuse-bridge.c:<wbr>3997<br>#2 0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>---Type <return> to continue, or q <return> to quit---<br>#3 0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 3 (LWP 25176):<br>#0 0x00007f1ead19dbf7 in readv () from ./lib64/libc.so.6<br>#1 0x00007f1eae59d3dd in sys_readv (fd=8, iov=0x7f1e9a278e20, iovcnt=2)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/<wbr>libglusterfs/src/syscall.c:295<br>#2 0x00007f1ea6313a02 in fuse_thread_proc (data=0xe1f910)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>mount/fuse/src/fuse-bridge.c:<wbr>4977<br>#3 0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#4 0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 2 (LWP 25172):<br>#0 0x00007f1ead4d8f41 in ?? () from ./usr/lib64/libcrypto.so.10<br>#1 0x00007f1ead4d909b in AES_encrypt () from ./usr/lib64/libcrypto.so.10<br>#2 0x00007f1e9400ad3c in ?? ()<br>#3 0x00007f1e9400addc in ?? ()<br>#4 0x00007f1ea16a7290 in ?? ()<br>#5 0x00007f1ea16a7238 in ?? ()<br>#6 0x0000000000000000 in ?? ()<br><br>Thread 1 (LWP 25174):<br>#0 0x00007f1ea056393a in offset_at_tail (conf=0xc0, object=0x7f1e9400ac28)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>encryption/crypt/src/atom.c:96<br>#1 0x00007f1ea05639e4 in offset_at_data_tail (frame=0x7f1e94001320, object=0x7f1e9400ac28)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>encryption/crypt/src/atom.c:<wbr>110<br>#2 0x00007f1ea0564738 in rmw_partial_block (frame=0x7f1e94001320, cookie=0x7f1e8c00c0b0, this=0x7f1e9c00b870, <br> op_ret=4096, op_errno=0, vec=0x7f1e8c00bd10, count=1, stbuf=0x7f1e8c017208, iobref=0x7f1e8c013900, <br> atom=0x7f1ea07846a0 <atoms+64>)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>encryption/crypt/src/atom.c:<wbr>523<br>#3 0x00007f1ea0565348 in rmw_data_tail (frame=0x7f1e94001320, cookie=0x7f1e8c00c0b0, this=0x7f1e9c00b870, <br> op_ret=4096, op_errno=0, vec=0x7f1e8c00bd10, count=1, stbuf=0x7f1e8c017208, iobref=0x7f1e8c013900, xdata=0x0)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>encryption/crypt/src/atom.c:<wbr>716<br>#4 0x00007f1ea0577699 in __crypt_readv_done (frame=0x7f1e8c00c0b0, cookie=0x0, this=0x7f1e9c00b870, op_ret=0, <br> op_errno=0, xdata=0x0)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>encryption/crypt/src/crypt.c:<wbr>3460<br>#5 0x00007f1ea0577774 in crypt_readv_done (frame=0x7f1e8c00c0b0, this=0x7f1e9c00b870)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>encryption/crypt/src/crypt.c:<wbr>3487<br>#6 0x00007f1ea0577b3a in put_one_call_readv (frame=0x7f1e8c00c0b0, this=0x7f1e9c00b870)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>encryption/crypt/src/crypt.c:<wbr>3514<br>#7 0x00007f1ea0566885 in crypt_readv_cbk (frame=0x7f1e8c00c0b0, cookie=0x7f1e8c00e8b0, this=0x7f1e9c00b870, <br> op_ret=4096, op_errno=0, vec=0x7f1e9ac79880, count=1, stbuf=0x7f1e9ac79810, iobref=0x7f1e8c013900, xdata=0x0)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>encryption/crypt/src/crypt.c:<wbr>371<br>#8 0x00007f1ea0810b7f in dht_readv_cbk (frame=0x7f1e8c00e8b0, cookie=0x7f1e8c003250, this=0x7f1e9c00a200, <br> op_ret=4096, op_errno=0, vector=0x7f1e9ac79880, count=1, stbuf=0x7f1e9ac79810, iobref=0x7f1e8c013900, xdata=0x0)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>cluster/dht/src/dht-inode-<wbr>read.c:479<br>#9 0x00007f1ea0a73f83 in client3_3_readv_cbk (req=0x7f1e8c0034c0, iov=0x7f1e8c003500, count=2, <br> myframe=0x7f1e8c003250)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/xlators/<wbr>protocol/client/src/client-<wbr>rpc-fops.c:2927<br>#10 0x00007f1eae32a84d in rpc_clnt_handle_reply (clnt=0x7f1e9c03ec30, pollin=0x7f1e8c002570)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/rpc/rpc-<wbr>lib/src/rpc-clnt.c:778<br>#11 0x00007f1eae32ae17 in rpc_clnt_notify (trans=0x7f1e9c03ee00, mydata=0x7f1e9c03ec60, <br> event=RPC_TRANSPORT_MSG_<wbr>RECEIVED, data=0x7f1e8c002570)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/rpc/rpc-<wbr>lib/src/rpc-clnt.c:971<br>#12 0x00007f1eae326dac in rpc_transport_notify (this=0x7f1e9c03ee00, event=RPC_TRANSPORT_MSG_<wbr>RECEIVED, <br> data=0x7f1e8c002570) at /home/jenkins/root/workspace/<wbr>centos6-regression/rpc/rpc-<wbr>lib/src/rpc-transport.c:538<br>#13 0x00007f1ea2ed858a in socket_event_poll_in (this=0x7f1e9c03ee00, notify_handled=_gf_true)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/rpc/rpc-<wbr>transport/socket/src/socket.c:<wbr>2315<br>#14 0x00007f1ea2ed8bd5 in socket_event_handler (fd=10, idx=2, gen=1, data=0x7f1e9c03ee00, poll_in=1, poll_out=0, <br> poll_err=0) at /home/jenkins/root/workspace/<wbr>centos6-regression/rpc/rpc-<wbr>transport/socket/src/socket.c:<wbr>2467<br>#15 0x00007f1eae5d6b42 in event_dispatch_epoll_handler (event_pool=0xe18050, event=0x7f1e9ac79e70)<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/<wbr>libglusterfs/src/event-epoll.<wbr>c:572<br>#16 0x00007f1eae5d6e44 in event_dispatch_epoll_worker (data=0x7f1e9c03a000)<br>---Type <return> to continue, or q <return> to quit---<br> at /home/jenkins/root/workspace/<wbr>centos6-regression/<wbr>libglusterfs/src/event-epoll.<wbr>c:648<br>#17 0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#18 0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br><br></div></div></div>
</blockquote></div><br></div>