<div dir="ltr"><div>fstat report: <a href="https://fstat.gluster.org/summary?start_date=2017-05-25&amp;end_date=2017-06-06">https://fstat.gluster.org/summary?start_date=2017-05-25&amp;end_date=2017-06-06</a><br><br>One of the regression regression link : <a href="https://build.gluster.org/job/centos6-regression/4908/console">https://build.gluster.org/job/centos6-regression/4908/console</a><br><br></div>Who can help here?<br><div><div><br><br>(gdb) t a a bt<br><br>Thread 11 (LWP 25166):<br>#0  0x00007f1ead84500d in nanosleep () from ./lib64/libpthread.so.0<br>#1  0x00007f1eae576365 in gf_timer_proc (data=0xe34ea0)<br>    at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/timer.c:164<br>#2  0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#3  0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 10 (LWP 25169):<br>#0  0x00007f1ead841a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from ./lib64/libpthread.so.0<br>#1  0x00007f1eae5b0f9c in syncenv_task (proc=0xe35690)<br>    at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:603<br>#2  0x00007f1eae5b123e in syncenv_processor (thdata=0xe35690)<br>    at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:695<br>#3  0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#4  0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 9 (LWP 25173):<br>#0  0x00007f1ead841a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from ./lib64/libpthread.so.0<br>#1  0x00007f1e9b9c5d7a in iot_worker (data=0x7f1e9c024ec0)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/performance/io-threads/src/io-threads.c:193<br>#2  0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#3  0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 8 (LWP 25170):<br>#0  0x00007f1ead841a5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from ./lib64/libpthread.so.0<br>#1  0x00007f1eae5b0f9c in syncenv_task (proc=0xe35a50)<br>    at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:603<br>#2  0x00007f1eae5b123e in syncenv_processor (thdata=0xe35a50)<br>    at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syncop.c:695<br>#3  0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#4  0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 7 (LWP 25168):<br>#0  0x00007f1ead169c4d in nanosleep () from ./lib64/libc.so.6<br>#1  0x00007f1ead169ac0 in sleep () from ./lib64/libc.so.6<br>#2  0x00007f1eae59b946 in pool_sweeper (arg=0x0)<br>    at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/mem-pool.c:465<br>#3  0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#4  0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 6 (LWP 25167):<br>#0  0x00007f1ead845585 in sigwait () from ./lib64/libpthread.so.0<br>#1  0x0000000000409f22 in glusterfs_sigwaiter (arg=0x7ffc45a51640)<br>    at /home/jenkins/root/workspace/centos6-regression/glusterfsd/src/glusterfsd.c:2111<br>#2  0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#3  0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 5 (LWP 25165):<br>#0  0x00007f1ead83e2fd in pthread_join () from ./lib64/libpthread.so.0<br>#1  0x00007f1eae5d706f in event_dispatch_epoll (event_pool=0xe18050)<br>    at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:732<br>#2  0x00007f1eae59a096 in event_dispatch (event_pool=0xe18050)<br>    at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event.c:124<br>#3  0x000000000040ac33 in main (argc=6, argv=0x7ffc45a528b8)<br>    at /home/jenkins/root/workspace/centos6-regression/glusterfsd/src/glusterfsd.c:2522<br><br>Thread 4 (LWP 25177):<br>#0  0x00007f1ead84168c in pthread_cond_wait@@GLIBC_2.3.2 () from ./lib64/libpthread.so.0<br>#1  0x00007f1ea63110b6 in notify_kernel_loop (data=0xe1f910)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/mount/fuse/src/fuse-bridge.c:3997<br>#2  0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit---<br>#3  0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 3 (LWP 25176):<br>#0  0x00007f1ead19dbf7 in readv () from ./lib64/libc.so.6<br>#1  0x00007f1eae59d3dd in sys_readv (fd=8, iov=0x7f1e9a278e20, iovcnt=2)<br>    at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/syscall.c:295<br>#2  0x00007f1ea6313a02 in fuse_thread_proc (data=0xe1f910)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/mount/fuse/src/fuse-bridge.c:4977<br>#3  0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#4  0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br>Thread 2 (LWP 25172):<br>#0  0x00007f1ead4d8f41 in ?? () from ./usr/lib64/libcrypto.so.10<br>#1  0x00007f1ead4d909b in AES_encrypt () from ./usr/lib64/libcrypto.so.10<br>#2  0x00007f1e9400ad3c in ?? ()<br>#3  0x00007f1e9400addc in ?? ()<br>#4  0x00007f1ea16a7290 in ?? ()<br>#5  0x00007f1ea16a7238 in ?? ()<br>#6  0x0000000000000000 in ?? ()<br><br>Thread 1 (LWP 25174):<br>#0  0x00007f1ea056393a in offset_at_tail (conf=0xc0, object=0x7f1e9400ac28)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/encryption/crypt/src/atom.c:96<br>#1  0x00007f1ea05639e4 in offset_at_data_tail (frame=0x7f1e94001320, object=0x7f1e9400ac28)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/encryption/crypt/src/atom.c:110<br>#2  0x00007f1ea0564738 in rmw_partial_block (frame=0x7f1e94001320, cookie=0x7f1e8c00c0b0, this=0x7f1e9c00b870, <br>    op_ret=4096, op_errno=0, vec=0x7f1e8c00bd10, count=1, stbuf=0x7f1e8c017208, iobref=0x7f1e8c013900, <br>    atom=0x7f1ea07846a0 &lt;atoms+64&gt;)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/encryption/crypt/src/atom.c:523<br>#3  0x00007f1ea0565348 in rmw_data_tail (frame=0x7f1e94001320, cookie=0x7f1e8c00c0b0, this=0x7f1e9c00b870, <br>    op_ret=4096, op_errno=0, vec=0x7f1e8c00bd10, count=1, stbuf=0x7f1e8c017208, iobref=0x7f1e8c013900, xdata=0x0)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/encryption/crypt/src/atom.c:716<br>#4  0x00007f1ea0577699 in __crypt_readv_done (frame=0x7f1e8c00c0b0, cookie=0x0, this=0x7f1e9c00b870, op_ret=0, <br>    op_errno=0, xdata=0x0)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/encryption/crypt/src/crypt.c:3460<br>#5  0x00007f1ea0577774 in crypt_readv_done (frame=0x7f1e8c00c0b0, this=0x7f1e9c00b870)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/encryption/crypt/src/crypt.c:3487<br>#6  0x00007f1ea0577b3a in put_one_call_readv (frame=0x7f1e8c00c0b0, this=0x7f1e9c00b870)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/encryption/crypt/src/crypt.c:3514<br>#7  0x00007f1ea0566885 in crypt_readv_cbk (frame=0x7f1e8c00c0b0, cookie=0x7f1e8c00e8b0, this=0x7f1e9c00b870, <br>    op_ret=4096, op_errno=0, vec=0x7f1e9ac79880, count=1, stbuf=0x7f1e9ac79810, iobref=0x7f1e8c013900, xdata=0x0)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/encryption/crypt/src/crypt.c:371<br>#8  0x00007f1ea0810b7f in dht_readv_cbk (frame=0x7f1e8c00e8b0, cookie=0x7f1e8c003250, this=0x7f1e9c00a200, <br>    op_ret=4096, op_errno=0, vector=0x7f1e9ac79880, count=1, stbuf=0x7f1e9ac79810, iobref=0x7f1e8c013900, xdata=0x0)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/cluster/dht/src/dht-inode-read.c:479<br>#9  0x00007f1ea0a73f83 in client3_3_readv_cbk (req=0x7f1e8c0034c0, iov=0x7f1e8c003500, count=2, <br>    myframe=0x7f1e8c003250)<br>    at /home/jenkins/root/workspace/centos6-regression/xlators/protocol/client/src/client-rpc-fops.c:2927<br>#10 0x00007f1eae32a84d in rpc_clnt_handle_reply (clnt=0x7f1e9c03ec30, pollin=0x7f1e8c002570)<br>    at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-lib/src/rpc-clnt.c:778<br>#11 0x00007f1eae32ae17 in rpc_clnt_notify (trans=0x7f1e9c03ee00, mydata=0x7f1e9c03ec60, <br>    event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f1e8c002570)<br>    at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-lib/src/rpc-clnt.c:971<br>#12 0x00007f1eae326dac in rpc_transport_notify (this=0x7f1e9c03ee00, event=RPC_TRANSPORT_MSG_RECEIVED, <br>    data=0x7f1e8c002570) at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-lib/src/rpc-transport.c:538<br>#13 0x00007f1ea2ed858a in socket_event_poll_in (this=0x7f1e9c03ee00, notify_handled=_gf_true)<br>    at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-transport/socket/src/socket.c:2315<br>#14 0x00007f1ea2ed8bd5 in socket_event_handler (fd=10, idx=2, gen=1, data=0x7f1e9c03ee00, poll_in=1, poll_out=0, <br>    poll_err=0) at /home/jenkins/root/workspace/centos6-regression/rpc/rpc-transport/socket/src/socket.c:2467<br>#15 0x00007f1eae5d6b42 in event_dispatch_epoll_handler (event_pool=0xe18050, event=0x7f1e9ac79e70)<br>    at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:572<br>#16 0x00007f1eae5d6e44 in event_dispatch_epoll_worker (data=0x7f1e9c03a000)<br>---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit---<br>    at /home/jenkins/root/workspace/centos6-regression/libglusterfs/src/event-epoll.c:648<br>#17 0x00007f1ead83daa1 in start_thread () from ./lib64/libpthread.so.0<br>#18 0x00007f1ead1a5bcd in clone () from ./lib64/libc.so.6<br><br><br></div></div></div>