[Bugs] [Bug 1662557] glusterfs process crashes, causing "Transport endpoint not connected".
bugzilla at redhat.com
bugzilla at redhat.com
Thu Jan 3 12:32:51 UTC 2019
https://bugzilla.redhat.com/show_bug.cgi?id=1662557
robdewit <rob.dewit at coosto.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Flags|needinfo?(rob.dewit at coosto. |
|com) |
--- Comment #7 from robdewit <rob.dewit at coosto.com> ---
Good question! It turns out this is not always the case. I checked some other
coredumps:
coredump 1 - same backtrace
coredump 2 - untraceable
coredump 3 - A different backtrace:
(gdb) bt
#0 0x00007f2f2a1d32c0 in ?? () from /lib64/libuuid.so.1
#1 0x00007f2f2a1d24e0 in uuid_compare () from /lib64/libuuid.so.1
#2 0x00007f2f2aa57691 in gf_uuid_compare (u2=0x7f2f115e67f8
"\270x\274\226Z\301F\006\256\221\230\005\031\321N\342\001",
u1=0x7f2eff90 <Address 0x7f2eff90 out of bounds>) at compat-uuid.h:25
#3 __inode_find (table=table at entry=0x7f2f20063b80,
gfid=gfid at entry=0x7f2f115e67f8
"\270x\274\226Z\301F\006\256\221\230\005\031\321N\342\001") at inode.c:892
#4 0x00007f2f2aa57d79 in inode_find (table=table at entry=0x7f2f20063b80,
gfid=gfid at entry=0x7f2f115e67f8
"\270x\274\226Z\301F\006\256\221\230\005\031\321N\342\001")
at inode.c:917
#5 0x00007f2f24a1ae72 in unserialize_rsp_direntp_v2 (this=0x7f2f2000e980,
fd=<optimized out>, rsp=rsp at entry=0x7f2f1e164a70, entries=0x7f2f1e164aa0)
at client-helpers.c:338
#6 0x00007f2f24a59005 in client_post_readdirp_v2 (this=<optimized out>,
rsp=0x7f2f1e164a70, fd=<optimized out>, entries=<optimized out>,
xdata=0x7f2f1e164a68)
at client-common.c:3533
#7 0x00007f2f24a6b226 in client4_0_readdirp_cbk (req=<optimized out>,
iov=0x7f2f0b99d508, count=<optimized out>, myframe=0x7f2ef4a691f8) at
client-rpc-fops_v2.c:2333
#8 0x00007f2f2a814534 in rpc_clnt_handle_reply
(clnt=clnt at entry=0x7f2f2004f530, pollin=pollin at entry=0x7f2f114a8290) at
rpc-clnt.c:755
#9 0x00007f2f2a814e77 in rpc_clnt_notify (trans=0x7f2f2004f860,
mydata=0x7f2f2004f560, event=<optimized out>, data=0x7f2f114a8290) at
rpc-clnt.c:923
#10 0x00007f2f2a810f13 in rpc_transport_notify (this=this at entry=0x7f2f2004f860,
event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f2f114a8290)
at rpc-transport.c:525
#11 0x00007f2f25928a23 in socket_event_poll_in (notify_handled=true,
this=0x7f2f2004f860) at socket.c:2504
#12 socket_event_handler (fd=290095760, idx=2, gen=4, data=0x7f2f2004f860,
poll_in=<optimized out>, poll_out=<optimized out>, poll_err=0) at socket.c:2905
#13 0x00007f2f2aaa9aeb in event_dispatch_epoll_handler (event=0x7f2f1e164ed0,
event_pool=0x7510b0) at event-epoll.c:591
#14 event_dispatch_epoll_worker (data=0x7f2f2004f310) at event-epoll.c:668
#15 0x00007f2f2989a504 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f2f2918219f in clone () from /lib64/libc.so.6
coredump 4 - yet another backtrace:
(gdb) bt
#0 0x00007ff2249a58a4 in _int_free () from /lib64/libc.so.6
#1 0x00007ff2249aac9e in free () from /lib64/libc.so.6
#2 0x00007ff22631d6af in __gf_free (free_ptr=<optimized out>) at
mem-pool.c:356
#3 0x00007ff223bf1410 in free_fuse_state (state=0x7ff1f4760430) at
fuse-helpers.c:81
#4 0x00007ff223bf70a9 in fuse_err_cbk (frame=0x7ff1f471b1d8, cookie=<optimized
out>, this=0x18dddb0, op_ret=0, op_errno=0, xdata=<optimized out>)
at fuse-bridge.c:1434
#5 0x00007ff21aebc29d in io_stats_flush_cbk (frame=0x7ff206a0b088,
cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=0, xdata=0x0)
at io-stats.c:2286
#6 0x00007ff226385b29 in default_flush_cbk (frame=0x7ff1f4737f58,
cookie=<optimized out>, this=<optimized out>, op_ret=0, op_errno=0, xdata=0x0)
at defaults.c:1159
#7 0x00007ff21b926f77 in ra_flush_cbk (frame=0x7ff1f4737238, cookie=<optimized
out>, this=<optimized out>, op_ret=0, op_errno=0, xdata=0x0) at
read-ahead.c:539
#8 0x00007ff21bb390dd in wb_flush_helper (frame=0x7ff2071e7488,
this=<optimized out>, fd=<optimized out>, xdata=0x0) at write-behind.c:1987
#9 0x00007ff22631a055 in call_resume_keep_stub (stub=0x7ff1f4744da8) at
call-stub.c:2563
#10 0x00007ff21bb3c999 in wb_do_winds (wb_inode=wb_inode at entry=0x7ff1f4742730,
tasks=tasks at entry=0x7ff220d42640) at write-behind.c:1737
#11 0x00007ff21bb3ca9c in wb_process_queue
(wb_inode=wb_inode at entry=0x7ff1f4742730) at write-behind.c:1778
#12 0x00007ff21bb41a07 in wb_fulfill_cbk (frame=frame at entry=0x7ff21d48e7c8,
cookie=<optimized out>, this=<optimized out>, op_ret=op_ret at entry=123,
op_errno=op_errno at entry=0, prebuf=prebuf at entry=0x7ff21d4ac610,
postbuf=postbuf at entry=0x7ff21d4ac6a8, xdata=xdata at entry=0x7ff21d490168) at
write-behind.c:1105
#13 0x00007ff21bdbde86 in dht_writev_cbk (frame=frame at entry=0x7ff21c08d7c8,
cookie=<optimized out>, this=<optimized out>, op_ret=123, op_errno=0,
prebuf=prebuf at entry=0x7ff21d4ac610, postbuf=postbuf at entry=0x7ff21d4ac6a8,
xdata=0x7ff21d490168) at dht-inode-write.c:140
#14 0x00007ff22003e21e in afr_writev_unwind (frame=frame at entry=0x7ff21d4a3ee8,
this=this at entry=0x7ff21c0110c0) at afr-inode-write.c:234
#15 0x00007ff22003e7e6 in afr_writev_wind_cbk (this=0x7ff21c0110c0,
frame=0x7ff21d49ab08, cookie=<optimized out>, op_ret=<optimized out>,
op_errno=<optimized out>,
prebuf=<optimized out>, postbuf=<optimized out>, xdata=<optimized out>) at
afr-inode-write.c:388
#16 afr_writev_wind_cbk (frame=0x7ff21d49ab08, cookie=<optimized out>,
this=0x7ff21c0110c0, op_ret=<optimized out>, op_errno=<optimized out>,
prebuf=<optimized out>,
postbuf=0x7ff220d42980, xdata=0x7ff21d49ae58) at afr-inode-write.c:354
#17 0x00007ff220313748 in client4_0_writev_cbk (req=<optimized out>,
iov=<optimized out>, count=<optimized out>, myframe=0x7ff21d483a58) at
client-rpc-fops_v2.c:685
#18 0x00007ff2260bf534 in rpc_clnt_handle_reply
(clnt=clnt at entry=0x7ff21c04f530, pollin=pollin at entry=0x7ff21d49e650) at
rpc-clnt.c:755
#19 0x00007ff2260bfe77 in rpc_clnt_notify (trans=0x7ff21c04f860,
mydata=0x7ff21c04f560, event=<optimized out>, data=0x7ff21d49e650) at
rpc-clnt.c:923
#20 0x00007ff2260bbf13 in rpc_transport_notify (this=this at entry=0x7ff21c04f860,
event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7ff21d49e650)
at rpc-transport.c:525
#21 0x00007ff2211d3a23 in socket_event_poll_in (notify_handled=true,
this=0x7ff21c04f860) at socket.c:2504
#22 socket_event_handler (fd=491382352, idx=2, gen=4, data=0x7ff21c04f860,
poll_in=<optimized out>, poll_out=<optimized out>, poll_err=0) at socket.c:2905
#23 0x00007ff226354aeb in event_dispatch_epoll_handler (event=0x7ff220d42ed0,
event_pool=0x18d70b0) at event-epoll.c:591
#24 event_dispatch_epoll_worker (data=0x1913840) at event-epoll.c:668
#25 0x00007ff225145504 in start_thread () from /lib64/libpthread.so.0
#26 0x00007ff224a2d19f in clone () from /lib64/libc.so.6
coredump 5 - Another one:
(gdb) bt
#0 0x00007fad93d3ce30 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x00007fad94eea73e in gf_log_set_log_buf_size (buf_size=buf_size at entry=0)
at logging.c:273
#2 0x00007fad94eea8df in gf_log_disable_suppression_before_exit
(ctx=0x1334010) at logging.c:444
#3 0x00007fad94ef0f94 in gf_print_trace (signum=11, ctx=0x1334010) at
common-utils.c:922
#4 <signal handler called>
#5 0x00007fad94f0fd52 in fd_destroy (bound=true, fd=0x7fad64f216c8) at
fd.c:478
#6 fd_unref (fd=0x7fad64f216c8) at fd.c:529
#7 0x00007fad8eeba0e8 in client_local_wipe (local=local at entry=0x7fad8a924358)
at client-helpers.c:124
#8 0x00007fad8ef161e0 in client4_0_finodelk_cbk (req=<optimized out>,
iov=<optimized out>, count=<optimized out>, myframe=<optimized out>)
at client-rpc-fops_v2.c:1398
#9 0x00007fad94cb4534 in rpc_clnt_handle_reply
(clnt=clnt at entry=0x7fad8804f530, pollin=pollin at entry=0x7fad8a917950) at
rpc-clnt.c:755
#10 0x00007fad94cb4e77 in rpc_clnt_notify (trans=0x7fad8804f860,
mydata=0x7fad8804f560, event=<optimized out>, data=0x7fad8a917950) at
rpc-clnt.c:923
#11 0x00007fad94cb0f13 in rpc_transport_notify (this=this at entry=0x7fad8804f860,
event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7fad8a917950)
at rpc-transport.c:525
#12 0x00007fad8fdc8a23 in socket_event_poll_in (notify_handled=true,
this=0x7fad8804f860) at socket.c:2504
#13 socket_event_handler (fd=-1970177712, idx=2, gen=4, data=0x7fad8804f860,
poll_in=<optimized out>, poll_out=<optimized out>, poll_err=0) at socket.c:2905
#14 0x00007fad94f49aeb in event_dispatch_epoll_handler (event=0x7fad8f937ed0,
event_pool=0x136b0b0) at event-epoll.c:591
#15 event_dispatch_epoll_worker (data=0x13a7840) at event-epoll.c:668
#16 0x00007fad93d3a504 in start_thread () from /lib64/libpthread.so.0
#17 0x00007fad9362219f in clone () from /lib64/libc.so.6
coredump 6 - And another:
(gdb) bt
#0 0x00007f3c6caace30 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x00007f3c6dc82795 in __gf_free (free_ptr=0x7f3c39e43cb0) at mem-pool.c:333
#2 0x00007f3c6dc65d90 in __inode_ctx_free (inode=inode at entry=0x7f3c39e435a8)
at inode.c:322
#3 0x00007f3c6dc66e12 in __inode_destroy (inode=0x7f3c39e435a8) at inode.c:338
#4 inode_table_prune (table=table at entry=0x7f3c58010950) at inode.c:1535
#5 0x00007f3c6dc671ec in inode_unref (inode=0x7f3c39e435a8) at inode.c:542
#6 0x00007f3c679dbf97 in afr_local_cleanup (local=0x7f3c39e1f3e8,
this=<optimized out>) at afr-common.c:1995
#7 0x00007f3c679b5ee1 in afr_transaction_done (frame=<optimized out>,
this=<optimized out>) at afr-transaction.c:369
#8 0x00007f3c679b97f1 in afr_unlock_common_cbk
(frame=frame at entry=0x7f3c3a11d168, this=this at entry=0x7f3c600110c0,
op_ret=op_ret at entry=0, xdata=0x0,
op_errno=<optimized out>, cookie=<optimized out>) at afr-lk-common.c:243
#9 0x00007f3c679b98ae in afr_unlock_entrylk_cbk (frame=0x7f3c3a11d168,
cookie=<optimized out>, this=0x7f3c600110c0, op_ret=0, op_errno=<optimized
out>,
xdata=<optimized out>) at afr-lk-common.c:366
#10 0x00007f3c67c857bd in client4_0_entrylk_cbk (req=<optimized out>,
iov=<optimized out>, count=<optimized out>, myframe=<optimized out>) at
client-rpc-fops_v2.c:1446
#11 0x00007f3c6da24534 in rpc_clnt_handle_reply
(clnt=clnt at entry=0x7f3c60058e20, pollin=pollin at entry=0x7f3c5a5898e0) at
rpc-clnt.c:755
#12 0x00007f3c6da24e77 in rpc_clnt_notify (trans=0x7f3c600590e0,
mydata=0x7f3c60058e50, event=<optimized out>, data=0x7f3c5a5898e0) at
rpc-clnt.c:923
#13 0x00007f3c6da20f13 in rpc_transport_notify (this=this at entry=0x7f3c600590e0,
event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f3c5a5898e0)
at rpc-transport.c:525
#14 0x00007f3c68b38a23 in socket_event_poll_in (notify_handled=true,
this=0x7f3c600590e0) at socket.c:2504
#15 socket_event_handler (fd=1515755744, idx=4, gen=1, data=0x7f3c600590e0,
poll_in=<optimized out>, poll_out=<optimized out>, poll_err=0) at socket.c:2905
#16 0x00007f3c6dcb9aeb in event_dispatch_epoll_handler (event=0x7f3c65c04ed0,
event_pool=0x81a0b0) at event-epoll.c:591
#17 event_dispatch_epoll_worker (data=0x7f3c60043ad0) at event-epoll.c:668
#18 0x00007f3c6caaa504 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f3c6c39219f in clone () from /lib64/libc.so.6
coredump 7 - ...
(gdb) bt
#0 0x00007f916b526b88 in list_add (head=0x7f91389ba228, new=0x7f91389b9d78) at
../../../../libglusterfs/src/list.h:31
#1 wb_set_invalidate (wb_inode=0x7f91389b9d10, set=<optimized out>) at
write-behind.c:246
#2 wb_fulfill_cbk (frame=frame at entry=0x7f91617a2208, cookie=<optimized out>,
this=<optimized out>, op_ret=op_ret at entry=811, op_errno=op_errno at entry=0,
prebuf=prebuf at entry=0x7f91617ade00, postbuf=postbuf at entry=0x7f91617ade98,
xdata=xdata at entry=0x7f9160484c38) at write-behind.c:1095
#3 0x00007f916b7a2e86 in dht_writev_cbk (frame=frame at entry=0x7f91617b8838,
cookie=<optimized out>, this=<optimized out>, op_ret=811, op_errno=0,
prebuf=prebuf at entry=0x7f91617ade00, postbuf=postbuf at entry=0x7f91617ade98,
xdata=0x7f9160484c38) at dht-inode-write.c:140
#4 0x00007f916ba0c21e in afr_writev_unwind (frame=frame at entry=0x7f916100d918,
this=this at entry=0x7f91640110c0) at afr-inode-write.c:234
#5 0x00007f916ba0c7e6 in afr_writev_wind_cbk (this=0x7f91640110c0,
frame=0x7f91604865e8, cookie=<optimized out>, op_ret=<optimized out>,
op_errno=<optimized out>,
prebuf=<optimized out>, postbuf=<optimized out>, xdata=<optimized out>) at
afr-inode-write.c:388
#6 afr_writev_wind_cbk (frame=0x7f91604865e8, cookie=<optimized out>,
this=0x7f91640110c0, op_ret=<optimized out>, op_errno=<optimized out>,
prebuf=<optimized out>,
postbuf=0x7f916946c980, xdata=0x7f91614414b8) at afr-inode-write.c:354
#7 0x00007f916bce1748 in client4_0_writev_cbk (req=<optimized out>,
iov=<optimized out>, count=<optimized out>, myframe=0x7f915cfa2f98) at
client-rpc-fops_v2.c:685
#8 0x00007f9171a8d534 in rpc_clnt_handle_reply
(clnt=clnt at entry=0x7f9164050110, pollin=pollin at entry=0x7f9160481290) at
rpc-clnt.c:755
#9 0x00007f9171a8de77 in rpc_clnt_notify (trans=0x7f91640503d0,
mydata=0x7f9164050140, event=<optimized out>, data=0x7f9160481290) at
rpc-clnt.c:923
#10 0x00007f9171a89f13 in rpc_transport_notify (this=this at entry=0x7f91640503d0,
event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f9160481290)
at rpc-transport.c:525
#11 0x00007f916cba1a23 in socket_event_poll_in (notify_handled=true,
this=0x7f91640503d0) at socket.c:2504
#12 socket_event_handler (fd=1615336080, idx=2, gen=4, data=0x7f91640503d0,
poll_in=<optimized out>, poll_out=<optimized out>, poll_err=0) at socket.c:2905
#13 0x00007f9171d22aeb in event_dispatch_epoll_handler (event=0x7f916946ced0,
event_pool=0x24db0b0) at event-epoll.c:591
#14 event_dispatch_epoll_worker (data=0x7f9164048dc0) at event-epoll.c:668
#15 0x00007f9170b13504 in start_thread () from /lib64/libpthread.so.0
#16 0x00007f91703fb19f in clone () from /lib64/libc.so.6
coredump 8 - (gdb) bt
#0 0x00007f24560fbe30 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x00007f24572a973e in gf_log_set_log_buf_size (buf_size=buf_size at entry=0)
at logging.c:273
#2 0x00007f24572a98df in gf_log_disable_suppression_before_exit (ctx=0x840010)
at logging.c:444
#3 0x00007f24572aff94 in gf_print_trace (signum=11, ctx=0x840010) at
common-utils.c:922
#4 <signal handler called>
#5 0x00007f24572ced52 in fd_destroy (bound=true, fd=0x7f24380d3f98) at
fd.c:478
#6 fd_unref (fd=0x7f24380d3f98) at fd.c:529
#7 0x00007f24512790e8 in client_local_wipe (local=local at entry=0x7f243c0ad548)
at client-helpers.c:124
#8 0x00007f24512d51e0 in client4_0_finodelk_cbk (req=<optimized out>,
iov=<optimized out>, count=<optimized out>, myframe=<optimized out>)
at client-rpc-fops_v2.c:1398
#9 0x00007f2457073534 in rpc_clnt_handle_reply
(clnt=clnt at entry=0x7f244c050110, pollin=pollin at entry=0x7f2444165d30) at
rpc-clnt.c:755
#10 0x00007f2457073e77 in rpc_clnt_notify (trans=0x7f244c0503d0,
mydata=0x7f244c050140, event=<optimized out>, data=0x7f2444165d30) at
rpc-clnt.c:923
#11 0x00007f245706ff13 in rpc_transport_notify (this=this at entry=0x7f244c0503d0,
event=event at entry=RPC_TRANSPORT_MSG_RECEIVED, data=data at entry=0x7f2444165d30)
at rpc-transport.c:525
#12 0x00007f2452187a23 in socket_event_poll_in (notify_handled=true,
this=0x7f244c0503d0) at socket.c:2504
#13 socket_event_handler (fd=1142316336, idx=2, gen=4, data=0x7f244c0503d0,
poll_in=<optimized out>, poll_out=<optimized out>, poll_err=0) at socket.c:2905
#14 0x00007f2457308aeb in event_dispatch_epoll_handler (event=0x7f244b1b7ed0,
event_pool=0x8770b0) at event-epoll.c:591
#15 event_dispatch_epoll_worker (data=0x7f244c043ad0) at event-epoll.c:668
#16 0x00007f24560f9504 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f24559e119f in clone () from /lib64/libc.so.6
If you really need the info - I have some 20 more coredumps, I suspect they all
have different traces...
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list