[Gluster-devel] GlusterFS 1.3 mainlin 2.5-patch-317 crashes when second afr-node goes down

Fri Jul 20 18:30:46 UTC 2007

Urban,
 is it possible to get the logs from a little before what you pasted? also,
was the machine you brought down both server and a client ?

avati

2007/7/20, Urban Loesch <ul at enas.net>:
>
> Hi Avati,
>
> still the Problem when I shut down the second server.
> Here the Log
> 2007-07-20 18:51:15 C [tcp.c:81:tcp_disconnect] server: connection
> disconnected
> 2007-07-20 18:51:15 E [client-protocol.c:336:client_protocol_xfer]
> data1-gluster2-ds: transport_submit failed
> 2007-07-20 18:51:15 W [client-protocol.c:4196:client_protocol_cleanup]
> data1-gluster2-ds: cleaning up state in transport object 0x8052bd8
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 C [common-utils.c:210:gf_print_trace]
> debug-backtrace: Got signal (11), printing backtrace
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/libglusterfs.so.0(gf_print_trace+0x2d)
> [0xb7edc32d]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /lib/i686/cmov/libc.so.6 [0xb7d98208]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/libglusterfs.so.0(dict_ref+0x29) [0xb7ed5989]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace:
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so [0xb752b018]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace:
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so [0xb752c185]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace:
> /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so[0xb7538b1f]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so
> [0xb753d1d5]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so
> [0xb754bef3]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace:
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so [0xb756203f]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace:
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so(notify+0x46b)
> [0xb756072b]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/libglusterfs.so.0(transport_notify+0x37)
> [0xb7edd837]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xd9)
> [0xb7ede359]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/libglusterfs.so.0(poll_iteration+0x1d)
> [0xb7edd90d]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: [glusterfsd] [0x80493ee]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /lib/i686/cmov/libc.so.6(__libc_start_main+0xe0)
> [0xb7d84030]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: [glusterfsd] [0x8048d81]
>
> And the backtrace:
> gluster1:/# gdb glusterfsd -c core.15573
> GNU gdb 6.4.90-debian
> Copyright (C) 2006 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> This GDB was configured as "i486-linux-gnu"...(no debugging symbols found)
> Using host libthread_db library "/lib/i686/cmov/libthread_db.so.1".
>
> Reading symbols from /usr/lib/libglusterfs.so.0...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/libglusterfs.so.0
> Reading symbols from /lib/i686/cmov/libdl.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/i686/cmov/libdl.so.2
> Reading symbols from /lib/i686/cmov/libpthread.so.0...(no debugging
> symbols found)...done.
> Loaded symbols for /lib/i686/cmov/libpthread.so.0
> Reading symbols from /lib/i686/cmov/libc.so.6...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/i686/cmov/libc.so.6
> Reading symbols from /lib/ld-linux.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/ld-linux.so.2
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/storage/posix.so...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/storage/posix.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so...(no debugging
> symbols found)...done.
> Loaded symbols for
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so...(no
> debugging symbols found)...done.
> Loaded symbols for
> /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so...
> (no debugging symbols found)...done.
> Loaded symbols for
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/transport/tcp/client.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/transport/tcp/client.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/scheduler/rr.so...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/scheduler/rr.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/transport/tcp/server.so...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/transport/tcp/server.so
> Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/libgcc_s.so.1
>
> Core was generated by
> `[glusterfsd]
> '.
> Program terminated with signal 11, Segmentation fault.
> #0  0xb7ebc430 in pthread_mutex_lock () from
> /lib/i686/cmov/libpthread.so.0
> (gdb) backtrace
> #0  0xb7ebc430 in pthread_mutex_lock () from
> /lib/i686/cmov/libpthread.so.0
> #1  0xb7ed5989 in dict_ref () from /usr/lib/libglusterfs.so.0
> #2  0xb752b018 in fini () from
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
> #3  0xb752c185 in mop_lock () from
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
> #4  0xb7538b1f in ?? () from
> /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so
> #5  0x081f1924 in ?? ()
> #6  0x081e2a08 in ?? ()
> #7  0x08051970 in ?? ()
> #8  0x00020000 in ?? ()
> #9  0x00000022 in ?? ()
> #10 0xbfadbdf0 in ?? ()
> #11 0xbfadbd48 in ?? ()
> #12 0xb753d1d5 in ?? () from
> /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so
> #13 0x081e2a08 in ?? ()
> #14 0x081ec2c8 in ?? ()
> #15 0x080518f8 in ?? ()
> #16 0x00020000 in ?? ()
> #17 0x00000022 in ?? ()
> #18 0xbfadbdf0 in ?? ()
> #19 0xbfadbd88 in ?? ()
> #20 0xb754bef3 in fini () from
> /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so
> Previous frame inner to this frame (corrupt stack?)
>
> Thanks and regards
> Urban
>
>
>

-- 
Anand V. Avati