[Gluster-users] [Gluster-devel] GlusterFS 3.3.1 client crash (signal received: 6)

Song gluster at 163.com
Fri Sep 6 02:16:48 UTC 2013 

It's a pity I don't know how to re-create the issue. While there are 1-2 crashed clients in total 120 clients every day.

Below is gdb result:

(gdb) where
#0  0x0000003267432885 in raise () from /lib64/libc.so.6
#1  0x0000003267434065 in abort () from /lib64/libc.so.6
#2  0x000000326746f7a7 in __libc_message () from /lib64/libc.so.6
#3  0x00000032674750c6 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007fc4f2847684 in mem_put (ptr=0x7fc4b0a4c03c) at mem-pool.c:559
#5  0x00007fc4f281cc9b in dict_destroy (this=0x7fc4f12cc5cc) at dict.c:397
#6  0x00007fc4ede24c30 in afr_local_cleanup (local=0x7fc4ce68ac20, this=<value optimized out>) at afr-common.c:848
#7  0x00007fc4ede2c0f1 in afr_lookup_done (frame=0x18d5ae4, cookie=0x0, this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value optimized out>, inode=0x18d5b20, 
    buf=0x7fffcb83ec50, xattr=0x7fc4f12e1818, postparent=0x7fffcb83ebe0) at afr-common.c:1881
#8  afr_lookup_cbk (frame=0x18d5ae4, cookie=0x0, this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value optimized out>, inode=0x18d5b20, buf=0x7fffcb83ec50, 
    xattr=0x7fc4f12e1818, postparent=0x7fffcb83ebe0) at afr-common.c:2044
#9  0x00007fc4ee066550 in client3_1_lookup_cbk (req=<value optimized out>, iov=<value optimized out>, count=<value optimized out>, myframe=0x7fc4f16f390c) at client3_1-fops.c:2636
#10 0x00007fc4f25ff4e5 in rpc_clnt_handle_reply (clnt=0x3b5c600, pollin=0x6ba00f0) at rpc-clnt.c:786
#11 0x00007fc4f25ffce0 in rpc_clnt_notify (trans=<value optimized out>, mydata=0x3b5c630, event=<value optimized out>, data=<value optimized out>) at rpc-clnt.c:905
#12 0x00007fc4f25faeb8 in rpc_transport_notify (this=<value optimized out>, event=<value optimized out>, data=<value optimized out>) at rpc-transport.c:489
#13 0x00007fc4eeeb0764 in socket_event_poll_in (this=0x3b6c060) at socket.c:1677
#14 0x00007fc4eeeb0847 in socket_event_handler (fd=<value optimized out>, idx=265, data=0x3b6c060, poll_in=1, poll_out=0, poll_err=<value optimized out>) at socket.c:1792
#15 0x00007fc4f2846464 in event_dispatch_epoll_handler (event_pool=0x177cdf0) at event.c:785
#16 event_dispatch_epoll (event_pool=0x177cdf0) at event.c:847
#17 0x000000000040736a in main (argc=<value optimized out>, argv=0x7fffcb83efc8) at glusterfsd.c:1689


-----Original Message-----
From: jowalker at redhat.com [mailto:jowalker at redhat.com] On Behalf Of John Mark Walker
Sent: Thursday, September 05, 2013 1:06 PM
To: Pranith Kumar Karampuri
Cc: Song; gluster-devel at nongnu.org
Subject: Re: [Gluster-devel] GlusterFS 3.3.1 client crash (signal received: 6)

Posting to gluster-users.


----- Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:
> Song,
> Seems like the issue is happening because of double 'memput', Could you let us know the steps to re-create the issue? Or the load that may lead to this?
> 
> Pranith
> 
> ----- Original Message -----
> > From: "Song" <gluster at 163.com>
> > To: gluster-devel at nongnu.org
> > Sent: Thursday, September 5, 2013 8:05:57 AM
> > Subject: [Gluster-devel] GlusterFS 3.3.1 client crash (signal 
> > received: 6)
> > 
> > 
> > 
> > I installed GlusterFS 3.3.1 in my 24 servers, created a DHT+AFR 
> > volume and mounted it with native client.
> > 
> > Recently, some glusterfs clients is crashed, the log is as below.
> > 
> > 
> > 
> > The OS is 64bit CentOS6.2, kernel version: 
> > 2.6.32-220.23.1.el6.x86_64 #1 SMP Fri Jun 28 00:56:49 CST 2013 
> > x86_64 x86_64 x86_64 GNU/Linux
> > 
> > 
> > 
> > 
> > 
> > pending frames:
> > 
> > frame : type(1) op(LOOKUP)
> > 
> > frame : type(1) op(LOOKUP)
> > 
> > frame : type(1) op(LOOKUP)
> > 
> > 
> > 
> > patchset: git://git.gluster.com/glusterfs.git
> > 
> > signal received: 6
> > 
> > time of crash: 2013-09-05 00:37:40
> > 
> > configuration details:
> > 
> > argp 1
> > 
> > backtrace 1
> > 
> > dlfcn 1
> > 
> > fdatasync 1
> > 
> > libpthread 1
> > 
> > llistxattr 1
> > 
> > setfsid 1
> > 
> > spinlock 1
> > 
> > epoll.h 1
> > 
> > xattr.h 1
> > 
> > st_atim.tv_nsec 1
> > 
> > package-string: glusterfs 3.3.1
> > 
> > /lib64/libc.so.6[0x3ac0232900]
> > 
> > /lib64/libc.so.6(gsignal+0x35)[0x3ac0232885]
> > 
> > /lib64/libc.so.6(abort+0x175)[0x3ac0234065]
> > 
> > /lib64/libc.so.6[0x3ac026f7a7]
> > 
> > /lib64/libc.so.6[0x3ac02750c6]
> > 
> > /usr/lib/libglusterfs.so.0(mem_put+0x64)[0x7f3f99c2c684]
> > 
> > /usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_local_clean
> > up+0x60)[0x7f3f95209c30]
> > 
> > /usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_lookup_cbk+
> > 0x5a1)[0x7f3f952110f1]
> > 
> > /usr/lib/glusterfs/3.3.1/xlator/protocol/client.so(client3_1_lookup_
> > cbk+0x6b0)[0x7f3f9544b550]
> > 
> > /usr/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7f3f999e44e5]
> > 
> > /usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f3f999e4ce0]
> > 
> > /usr/lib/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f3f999dfeb8]
> > 
> > /usr/lib/glusterfs/3.3.1/rpc-transport/socket.so(socket_event_poll_i
> > n+0x34)[0x7f3f96295764]
> > 
> > /usr/lib/glusterfs/3.3.1/rpc-transport/socket.so(socket_event_handle
> > r+0xc7)[0x7f3f96295847]
> > 
> > /usr/lib/libglusterfs.so.0(+0x3e464)[0x7f3f99c2b464]
> > 
> > /usr/sbin/glusterfs(main+0x58a)[0x40736a]
> > 
> > /lib64/libc.so.6(__libc_start_main+0xfd)[0x3ac021ecdd]
> > 
> > /usr/sbin/glusterfs[0x4042d9]
> > 
> > ---------
> > 
> > 
> > 
> > Best regards.
> > 
> > Willard Song
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at nongnu.org
> > https://lists.nongnu.org/mailman/listinfo/gluster-devel
> > 
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> https://lists.nongnu.org/mailman/listinfo/gluster-devel

More information about the Gluster-users mailing list