[Gluster-users] [Gluster-devel] GlusterFS 3.3.1 client crash (signal received: 6)

Pranith Kumar Karampuri pkarampu at redhat.com
Fri Sep 6 08:01:23 UTC 2013 

Song,
     Do one thing. Raise a bug with this backtrace. We shall see what we can do. Try to attach the full mount's log when the next crash happens. We shall check if that gives us any more info to debug the issue further.
Thanks for reporting the issue.

Pranith.
----- Original Message -----
> From: "Song" <gluster at 163.com>
> To: "John Mark Walker" <johnmark at gluster.org>, "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> Cc: gluster-users at gluster.org
> Sent: Friday, September 6, 2013 7:46:48 AM
> Subject: RE: [Gluster-devel] GlusterFS 3.3.1 client crash (signal received: 6)
> 
> It's a pity I don't know how to re-create the issue. While there are 1-2
> crashed clients in total 120 clients every day.
> 
> Below is gdb result:
> 
> (gdb) where
> #0  0x0000003267432885 in raise () from /lib64/libc.so.6
> #1  0x0000003267434065 in abort () from /lib64/libc.so.6
> #2  0x000000326746f7a7 in __libc_message () from /lib64/libc.so.6
> #3  0x00000032674750c6 in malloc_printerr () from /lib64/libc.so.6
> #4  0x00007fc4f2847684 in mem_put (ptr=0x7fc4b0a4c03c) at mem-pool.c:559
> #5  0x00007fc4f281cc9b in dict_destroy (this=0x7fc4f12cc5cc) at dict.c:397
> #6  0x00007fc4ede24c30 in afr_local_cleanup (local=0x7fc4ce68ac20,
> this=<value optimized out>) at afr-common.c:848
> #7  0x00007fc4ede2c0f1 in afr_lookup_done (frame=0x18d5ae4, cookie=0x0,
> this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value
> optimized out>, inode=0x18d5b20,
>     buf=0x7fffcb83ec50, xattr=0x7fc4f12e1818, postparent=0x7fffcb83ebe0) at
>     afr-common.c:1881
> #8  afr_lookup_cbk (frame=0x18d5ae4, cookie=0x0, this=<value optimized out>,
> op_ret=<value optimized out>, op_errno=<value optimized out>,
> inode=0x18d5b20, buf=0x7fffcb83ec50,
>     xattr=0x7fc4f12e1818, postparent=0x7fffcb83ebe0) at afr-common.c:2044
> #9  0x00007fc4ee066550 in client3_1_lookup_cbk (req=<value optimized out>,
> iov=<value optimized out>, count=<value optimized out>,
> myframe=0x7fc4f16f390c) at client3_1-fops.c:2636
> #10 0x00007fc4f25ff4e5 in rpc_clnt_handle_reply (clnt=0x3b5c600,
> pollin=0x6ba00f0) at rpc-clnt.c:786
> #11 0x00007fc4f25ffce0 in rpc_clnt_notify (trans=<value optimized out>,
> mydata=0x3b5c630, event=<value optimized out>, data=<value optimized out>)
> at rpc-clnt.c:905
> #12 0x00007fc4f25faeb8 in rpc_transport_notify (this=<value optimized out>,
> event=<value optimized out>, data=<value optimized out>) at
> rpc-transport.c:489
> #13 0x00007fc4eeeb0764 in socket_event_poll_in (this=0x3b6c060) at
> socket.c:1677
> #14 0x00007fc4eeeb0847 in socket_event_handler (fd=<value optimized out>,
> idx=265, data=0x3b6c060, poll_in=1, poll_out=0, poll_err=<value optimized
> out>) at socket.c:1792
> #15 0x00007fc4f2846464 in event_dispatch_epoll_handler (event_pool=0x177cdf0)
> at event.c:785
> #16 event_dispatch_epoll (event_pool=0x177cdf0) at event.c:847
> #17 0x000000000040736a in main (argc=<value optimized out>,
> argv=0x7fffcb83efc8) at glusterfsd.c:1689
> 
> 
> -----Original Message-----
> From: jowalker at redhat.com [mailto:jowalker at redhat.com] On Behalf Of John Mark
> Walker
> Sent: Thursday, September 05, 2013 1:06 PM
> To: Pranith Kumar Karampuri
> Cc: Song; gluster-devel at nongnu.org
> Subject: Re: [Gluster-devel] GlusterFS 3.3.1 client crash (signal received:
> 6)
> 
> Posting to gluster-users.
> 
> 
> ----- Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:
> > Song,
> > Seems like the issue is happening because of double 'memput', Could you let
> > us know the steps to re-create the issue? Or the load that may lead to
> > this?
> > 
> > Pranith
> > 
> > ----- Original Message -----
> > > From: "Song" <gluster at 163.com>
> > > To: gluster-devel at nongnu.org
> > > Sent: Thursday, September 5, 2013 8:05:57 AM
> > > Subject: [Gluster-devel] GlusterFS 3.3.1 client crash (signal
> > > received: 6)
> > > 
> > > 
> > > 
> > > I installed GlusterFS 3.3.1 in my 24 servers, created a DHT+AFR
> > > volume and mounted it with native client.
> > > 
> > > Recently, some glusterfs clients is crashed, the log is as below.
> > > 
> > > 
> > > 
> > > The OS is 64bit CentOS6.2, kernel version:
> > > 2.6.32-220.23.1.el6.x86_64 #1 SMP Fri Jun 28 00:56:49 CST 2013
> > > x86_64 x86_64 x86_64 GNU/Linux
> > > 
> > > 
> > > 
> > > 
> > > 
> > > pending frames:
> > > 
> > > frame : type(1) op(LOOKUP)
> > > 
> > > frame : type(1) op(LOOKUP)
> > > 
> > > frame : type(1) op(LOOKUP)
> > > 
> > > 
> > > 
> > > patchset: git://git.gluster.com/glusterfs.git
> > > 
> > > signal received: 6
> > > 
> > > time of crash: 2013-09-05 00:37:40
> > > 
> > > configuration details:
> > > 
> > > argp 1
> > > 
> > > backtrace 1
> > > 
> > > dlfcn 1
> > > 
> > > fdatasync 1
> > > 
> > > libpthread 1
> > > 
> > > llistxattr 1
> > > 
> > > setfsid 1
> > > 
> > > spinlock 1
> > > 
> > > epoll.h 1
> > > 
> > > xattr.h 1
> > > 
> > > st_atim.tv_nsec 1
> > > 
> > > package-string: glusterfs 3.3.1
> > > 
> > > /lib64/libc.so.6[0x3ac0232900]
> > > 
> > > /lib64/libc.so.6(gsignal+0x35)[0x3ac0232885]
> > > 
> > > /lib64/libc.so.6(abort+0x175)[0x3ac0234065]
> > > 
> > > /lib64/libc.so.6[0x3ac026f7a7]
> > > 
> > > /lib64/libc.so.6[0x3ac02750c6]
> > > 
> > > /usr/lib/libglusterfs.so.0(mem_put+0x64)[0x7f3f99c2c684]
> > > 
> > > /usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_local_clean
> > > up+0x60)[0x7f3f95209c30]
> > > 
> > > /usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_lookup_cbk+
> > > 0x5a1)[0x7f3f952110f1]
> > > 
> > > /usr/lib/glusterfs/3.3.1/xlator/protocol/client.so(client3_1_lookup_
> > > cbk+0x6b0)[0x7f3f9544b550]
> > > 
> > > /usr/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7f3f999e44e5]
> > > 
> > > /usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f3f999e4ce0]
> > > 
> > > /usr/lib/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f3f999dfeb8]
> > > 
> > > /usr/lib/glusterfs/3.3.1/rpc-transport/socket.so(socket_event_poll_i
> > > n+0x34)[0x7f3f96295764]
> > > 
> > > /usr/lib/glusterfs/3.3.1/rpc-transport/socket.so(socket_event_handle
> > > r+0xc7)[0x7f3f96295847]
> > > 
> > > /usr/lib/libglusterfs.so.0(+0x3e464)[0x7f3f99c2b464]
> > > 
> > > /usr/sbin/glusterfs(main+0x58a)[0x40736a]
> > > 
> > > /lib64/libc.so.6(__libc_start_main+0xfd)[0x3ac021ecdd]
> > > 
> > > /usr/sbin/glusterfs[0x4042d9]
> > > 
> > > ---------
> > > 
> > > 
> > > 
> > > Best regards.
> > > 
> > > Willard Song
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > _______________________________________________
> > > Gluster-devel mailing list
> > > Gluster-devel at nongnu.org
> > > https://lists.nongnu.org/mailman/listinfo/gluster-devel
> > > 
> > 
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at nongnu.org
> > https://lists.nongnu.org/mailman/listinfo/gluster-devel
> 
> 
>

More information about the Gluster-users mailing list