[Bugs] [Bug 1539680] New: RDMA transport bricks crash

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1539680

            Bug ID: 1539680
           Summary: RDMA transport bricks crash
           Product: GlusterFS
           Version: 3.12
         Component: rdma
          Severity: urgent
          Assignee: bugs at gluster.org
          Reporter: jiri.lunacek at hosting90.cz
                CC: bugs at gluster.org



Created attachment 1387736
  --> https://bugzilla.redhat.com/attachment.cgi?id=1387736&action=edit
brick volume file

Description of problem:
RDMA transport bricks crash

Version-Release number of selected component (if applicable):
glusterfs-client-xlators-3.12.4-1.el7.x86_64
glusterfs-api-3.12.4-1.el7.x86_64
glusterfs-rdma-3.12.4-1.el7.x86_64
glusterfs-libs-3.12.4-1.el7.x86_64
glusterfs-cli-3.12.4-1.el7.x86_64
glusterfs-server-3.12.4-1.el7.x86_64
glusterfs-fuse-3.12.4-1.el7.x86_64
glusterfs-3.12.4-1.el7.x86_64

How reproducible:
We are experiencing crashes of glusterfsd on bricks of a replicated RDMA
volume.
It may be interesting, that the two replicated bricks failed with the same
error several minutes one after another and are running after restart.
This happens approximately every 5 or 6 days.

/usr/lib64/glusterfs/3.12.4/rpc-transport/rdma.so(+0x467f) is always on top of
the stack.

Backtrace:
pending frames:
frame : type(0) op(12)
frame : type(0) op(12)
frame : type(0) op(29)
frame : type(0) op(37)
frame : type(0) op(29)
frame : type(0) op(29)
frame : type(0) op(29)
frame : type(0) op(29)
frame : type(0) op(29)
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash:
2018-01-29 11:58:11
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.12.4
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xa0)[0x7f3a13cc3500]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7f3a13ccd434]
/lib64/libc.so.6(+0x35270)[0x7f3a1232c270]
/usr/lib64/glusterfs/3.12.4/rpc-transport/rdma.so(+0x467f)[0x7f39fe8e867f]
/usr/lib64/glusterfs/3.12.4/rpc-transport/rdma.so(+0x48af)[0x7f39fe8e88af]
/usr/lib64/glusterfs/3.12.4/rpc-transport/rdma.so(__gf_rdma_do_gf_rdma_write+0x7d)[0x7f39fe8ec84d]
/usr/lib64/glusterfs/3.12.4/rpc-transport/rdma.so(__gf_rdma_send_reply_type_msg+0x184)[0x7f39fe8ecee4]
/usr/lib64/glusterfs/3.12.4/rpc-transport/rdma.so(__gf_rdma_ioq_churn_reply+0x128)[0x7f39fe8ed418]
/usr/lib64/glusterfs/3.12.4/rpc-transport/rdma.so(__gf_rdma_ioq_churn_entry+0x85)[0x7f39fe8ed665]
/usr/lib64/glusterfs/3.12.4/rpc-transport/rdma.so(+0xa2e0)[0x7f39fe8ee2e0]
/usr/lib64/glusterfs/3.12.4/rpc-transport/rdma.so(gf_rdma_submit_reply+0x96)[0x7f39fe8ee9b6]
/lib64/libgfrpc.so.0(rpcsvc_transport_submit+0x82)[0x7f3a13a832b2]
/lib64/libgfrpc.so.0(rpcsvc_submit_generic+0x180)[0x7f3a13a84f00]
/usr/lib64/glusterfs/3.12.4/xlator/protocol/server.so(+0x91cc)[0x7f39fef0b1cc]
/usr/lib64/glusterfs/3.12.4/xlator/protocol/server.so(+0x206f4)[0x7f39fef226f4]
/usr/lib64/glusterfs/3.12.4/xlator/debug/io-stats.so(+0x153a3)[0x7f39ff37a3a3]
/lib64/libglusterfs.so.0(default_readv_cbk+0x17b)[0x7f3a13d43deb]
/usr/lib64/glusterfs/3.12.4/xlator/features/upcall.so(+0x6cd1)[0x7f3a043e2cd1]
/usr/lib64/glusterfs/3.12.4/xlator/features/leases.so(+0x2a6b)[0x7f3a045f9a6b]
/usr/lib64/glusterfs/3.12.4/xlator/features/locks.so(+0x1025e)[0x7f3a04c3325e]
/usr/lib64/glusterfs/3.12.4/xlator/features/changetimerecorder.so(+0xdc14)[0x7f3a05b8bc14]
/usr/lib64/glusterfs/3.12.4/xlator/storage/posix.so(+0xf3d4)[0x7f3a065d93d4]
/lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3a13d40101]
/usr/lib64/glusterfs/3.12.4/xlator/features/changetimerecorder.so(+0x8daf)[0x7f3a05b86daf]
/lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3a13d40101]
/usr/lib64/glusterfs/3.12.4/xlator/features/bitrot-stub.so(+0xd9b1)[0x7f3a050749b1]
/lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3a13d40101]
/usr/lib64/glusterfs/3.12.4/xlator/features/locks.so(+0x190ec)[0x7f3a04c3c0ec]
/lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3a13d40101]
/lib64/libglusterfs.so.0(default_readv+0xe1)[0x7f3a13d40101]
/usr/lib64/glusterfs/3.12.4/xlator/features/leases.so(+0x6531)[0x7f3a045fd531]
/usr/lib64/glusterfs/3.12.4/xlator/features/upcall.so(+0x101ba)[0x7f3a043ec1ba]
/lib64/libglusterfs.so.0(default_readv_resume+0x1f3)[0x7f3a13d5a9c3]
/lib64/libglusterfs.so.0(call_resume_wind+0x2da)[0x7f3a13ce78ca]
/lib64/libglusterfs.so.0(call_resume+0x75)[0x7f3a13ce7df5]
/usr/lib64/glusterfs/3.12.4/xlator/performance/io-threads.so(+0x4de4)[0x7f3a041d5de4]
/lib64/libpthread.so.0(+0x7e25)[0x7f3a12b22e25]
/lib64/libc.so.6(clone+0x6d)[0x7f3a123ef34d]

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.