[Gluster-users] [Gluster-devel] GlusterFS 3.3.1 client crash (signal received: 6)

Song gluster at 163.com
Tue Oct 22 08:26:48 UTC 2013 

To locate this issue, is it possible to print more useful information in
backtrace?
When client crashed, trace information was printed. Which is coded in
function of "gf_print_trace", in common-utils.c.
I hope that some helpful debug information would be appended in this
function and when client crash next time, the data can help us to analyze
the problem.

Could you give me the suggestion what codes is useful?
Thanks!

-----Original Message-----
From: gluster-users-bounces at gluster.org
[mailto:gluster-users-bounces at gluster.org] On Behalf Of Song
Sent: Friday, September 06, 2013 10:17 AM
To: 'John Mark Walker'; 'Pranith Kumar Karampuri'
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] [Gluster-devel] GlusterFS 3.3.1 client crash
(signal received: 6)

It's a pity I don't know how to re-create the issue. While there are 1-2
crashed clients in total 120 clients every day.

Below is gdb result:

(gdb) where
#0  0x0000003267432885 in raise () from /lib64/libc.so.6
#1  0x0000003267434065 in abort () from /lib64/libc.so.6
#2  0x000000326746f7a7 in __libc_message () from /lib64/libc.so.6
#3  0x00000032674750c6 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007fc4f2847684 in mem_put (ptr=0x7fc4b0a4c03c) at mem-pool.c:559
#5  0x00007fc4f281cc9b in dict_destroy (this=0x7fc4f12cc5cc) at dict.c:397
#6  0x00007fc4ede24c30 in afr_local_cleanup (local=0x7fc4ce68ac20,
this=<value optimized out>) at afr-common.c:848
#7  0x00007fc4ede2c0f1 in afr_lookup_done (frame=0x18d5ae4, cookie=0x0,
this=<value optimized out>, op_ret=<value optimized out>, op_errno=<value
optimized out>, inode=0x18d5b20, 
    buf=0x7fffcb83ec50, xattr=0x7fc4f12e1818, postparent=0x7fffcb83ebe0) at
afr-common.c:1881
#8  afr_lookup_cbk (frame=0x18d5ae4, cookie=0x0, this=<value optimized out>,
op_ret=<value optimized out>, op_errno=<value optimized out>,
inode=0x18d5b20, buf=0x7fffcb83ec50, 
    xattr=0x7fc4f12e1818, postparent=0x7fffcb83ebe0) at afr-common.c:2044
#9  0x00007fc4ee066550 in client3_1_lookup_cbk (req=<value optimized out>,
iov=<value optimized out>, count=<value optimized out>,
myframe=0x7fc4f16f390c) at client3_1-fops.c:2636
#10 0x00007fc4f25ff4e5 in rpc_clnt_handle_reply (clnt=0x3b5c600,
pollin=0x6ba00f0) at rpc-clnt.c:786
#11 0x00007fc4f25ffce0 in rpc_clnt_notify (trans=<value optimized out>,
mydata=0x3b5c630, event=<value optimized out>, data=<value optimized out>)
at rpc-clnt.c:905
#12 0x00007fc4f25faeb8 in rpc_transport_notify (this=<value optimized out>,
event=<value optimized out>, data=<value optimized out>) at
rpc-transport.c:489
#13 0x00007fc4eeeb0764 in socket_event_poll_in (this=0x3b6c060) at
socket.c:1677
#14 0x00007fc4eeeb0847 in socket_event_handler (fd=<value optimized out>,
idx=265, data=0x3b6c060, poll_in=1, poll_out=0, poll_err=<value optimized
out>) at socket.c:1792
#15 0x00007fc4f2846464 in event_dispatch_epoll_handler
(event_pool=0x177cdf0) at event.c:785
#16 event_dispatch_epoll (event_pool=0x177cdf0) at event.c:847
#17 0x000000000040736a in main (argc=<value optimized out>,
argv=0x7fffcb83efc8) at glusterfsd.c:1689


-----Original Message-----
From: jowalker at redhat.com [mailto:jowalker at redhat.com] On Behalf Of John
Mark Walker
Sent: Thursday, September 05, 2013 1:06 PM
To: Pranith Kumar Karampuri
Cc: Song; gluster-devel at nongnu.org
Subject: Re: [Gluster-devel] GlusterFS 3.3.1 client crash (signal received:
6)

Posting to gluster-users.


----- Pranith Kumar Karampuri <pkarampu at redhat.com> wrote:
> Song,
> Seems like the issue is happening because of double 'memput', Could you
let us know the steps to re-create the issue? Or the load that may lead to
this?
> 
> Pranith
> 
> ----- Original Message -----
> > From: "Song" <gluster at 163.com>
> > To: gluster-devel at nongnu.org
> > Sent: Thursday, September 5, 2013 8:05:57 AM
> > Subject: [Gluster-devel] GlusterFS 3.3.1 client crash (signal
> > received: 6)
> > 
> > 
> > 
> > I installed GlusterFS 3.3.1 in my 24 servers, created a DHT+AFR 
> > volume and mounted it with native client.
> > 
> > Recently, some glusterfs clients is crashed, the log is as below.
> > 
> > 
> > 
> > The OS is 64bit CentOS6.2, kernel version: 
> > 2.6.32-220.23.1.el6.x86_64 #1 SMP Fri Jun 28 00:56:49 CST 2013
> > x86_64 x86_64 x86_64 GNU/Linux
> > 
> > 
> > 
> > 
> > 
> > pending frames:
> > 
> > frame : type(1) op(LOOKUP)
> > 
> > frame : type(1) op(LOOKUP)
> > 
> > frame : type(1) op(LOOKUP)
> > 
> > 
> > 
> > patchset: git://git.gluster.com/glusterfs.git
> > 
> > signal received: 6
> > 
> > time of crash: 2013-09-05 00:37:40
> > 
> > configuration details:
> > 
> > argp 1
> > 
> > backtrace 1
> > 
> > dlfcn 1
> > 
> > fdatasync 1
> > 
> > libpthread 1
> > 
> > llistxattr 1
> > 
> > setfsid 1
> > 
> > spinlock 1
> > 
> > epoll.h 1
> > 
> > xattr.h 1
> > 
> > st_atim.tv_nsec 1
> > 
> > package-string: glusterfs 3.3.1
> > 
> > /lib64/libc.so.6[0x3ac0232900]
> > 
> > /lib64/libc.so.6(gsignal+0x35)[0x3ac0232885]
> > 
> > /lib64/libc.so.6(abort+0x175)[0x3ac0234065]
> > 
> > /lib64/libc.so.6[0x3ac026f7a7]
> > 
> > /lib64/libc.so.6[0x3ac02750c6]
> > 
> > /usr/lib/libglusterfs.so.0(mem_put+0x64)[0x7f3f99c2c684]
> > 
> > /usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_local_clean
> > up+0x60)[0x7f3f95209c30]
> > 
> > /usr/lib/glusterfs/3.3.1/xlator/cluster/replicate.so(afr_lookup_cbk+
> > 0x5a1)[0x7f3f952110f1]
> > 
> > /usr/lib/glusterfs/3.3.1/xlator/protocol/client.so(client3_1_lookup_
> > cbk+0x6b0)[0x7f3f9544b550]
> > 
> > /usr/lib/libgfrpc.so.0(rpc_clnt_handle_reply+0xa5)[0x7f3f999e44e5]
> > 
> > /usr/lib/libgfrpc.so.0(rpc_clnt_notify+0x120)[0x7f3f999e4ce0]
> > 
> > /usr/lib/libgfrpc.so.0(rpc_transport_notify+0x28)[0x7f3f999dfeb8]
> > 
> > /usr/lib/glusterfs/3.3.1/rpc-transport/socket.so(socket_event_poll_i
> > n+0x34)[0x7f3f96295764]
> > 
> > /usr/lib/glusterfs/3.3.1/rpc-transport/socket.so(socket_event_handle
> > r+0xc7)[0x7f3f96295847]
> > 
> > /usr/lib/libglusterfs.so.0(+0x3e464)[0x7f3f99c2b464]
> > 
> > /usr/sbin/glusterfs(main+0x58a)[0x40736a]
> > 
> > /lib64/libc.so.6(__libc_start_main+0xfd)[0x3ac021ecdd]
> > 
> > /usr/sbin/glusterfs[0x4042d9]
> > 
> > ---------
> > 
> > 
> > 
> > Best regards.
> > 
> > Willard Song
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at nongnu.org
> > https://lists.nongnu.org/mailman/listinfo/gluster-devel
> > 
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> https://lists.nongnu.org/mailman/listinfo/gluster-devel


_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://supercolony.gluster.org/mailman/listinfo/gluster-users

More information about the Gluster-users mailing list