[Gluster-devel] GlusterFS 1.3 mainlin 2.5-patch-317 crashes when second afr-node goes down
Urban Loesch
ul at enas.net
Mon Jul 23 11:01:32 UTC 2007
Hi Avati,
today i found some difference in date & time on the three machines. I
correct the issue, so that all 3 machines have the same date and time.
It seems to work now, I will make some further tests and if I have
problems I will send you more logs.
My Setup uses 3 machines. 2 Servers an one seperate client. Server and
client are not the same machine.
Thanks
Urban
Anand Avati wrote:
> Urban,
> is it possible to get the logs from a little before what you pasted?
> also, was the machine you brought down both server and a client ?
>
> avati
>
> 2007/7/20, Urban Loesch < ul at enas.net <mailto:ul at enas.net>>:
>
> Hi Avati,
>
> still the Problem when I shut down the second server.
> Here the Log
> 2007-07-20 18:51:15 C [tcp.c:81:tcp_disconnect] server: connection
> disconnected
> 2007-07-20 18:51:15 E [client-protocol.c:336:client_protocol_xfer]
> data1-gluster2-ds: transport_submit failed
> 2007-07-20 18:51:15 W [client-protocol.c:4196:client_protocol_cleanup]
> data1-gluster2-ds: cleaning up state in transport object 0x8052bd8
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 W [client-protocol.c:4245:client_protocol_cleanup]
> data1-gluster2-ds: forced unwinding frame type(0) op(14)
> 2007-07-20 18:51:15 E [afr.c:1656:afr_writev_cbk] data1-ds-afr:
> (path=/ddump child=data1-gluster2-ds) op_ret=-1 op_errno=107
> 2007-07-20 18:51:15 C [common-utils.c:210:gf_print_trace]
> debug-backtrace: Got signal (11), printing backtrace
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/libglusterfs.so.0(gf_print_trace+0x2d)
> [0xb7edc32d]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /lib/i686/cmov/libc.so.6 [0xb7d98208]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/libglusterfs.so.0(dict_ref+0x29)
> [0xb7ed5989]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace:
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so [0xb752b018]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace:
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so [0xb752c185]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace:
> /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so
> [0xb7538b1f]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace:
> /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so
> [0xb753d1d5]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so
> [0xb754bef3]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace:
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so [0xb756203f]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace:
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so(notify+0x46b)
> [0xb756072b]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/libglusterfs.so.0(transport_notify+0x37)
> [0xb7edd837]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xd9)
> [0xb7ede359]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: /usr/lib/libglusterfs.so.0(poll_iteration+0x1d)
> [0xb7edd90d]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: [glusterfsd] [0x80493ee]
> 2007-07-20 18:51:15 C [common-utils.c :212:gf_print_trace]
> debug-backtrace: /lib/i686/cmov/libc.so.6(__libc_start_main+0xe0)
> [0xb7d84030]
> 2007-07-20 18:51:15 C [common-utils.c:212:gf_print_trace]
> debug-backtrace: [glusterfsd] [0x8048d81]
>
> And the backtrace:
> gluster1:/# gdb glusterfsd -c core.15573
> GNU gdb 6.4.90-debian
> Copyright (C) 2006 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License,
> and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB. Type "show warranty" for
> details.
> This GDB was configured as "i486-linux-gnu"...(no debugging
> symbols found)
> Using host libthread_db library "/lib/i686/cmov/libthread_db.so.1".
>
> Reading symbols from /usr/lib/libglusterfs.so.0...(no debugging
> symbols
> found)...done.
> Loaded symbols for /usr/lib/libglusterfs.so.0
> Reading symbols from /lib/i686/cmov/libdl.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/i686/cmov/libdl.so.2
> Reading symbols from /lib/i686/cmov/libpthread.so.0...(no debugging
> symbols found)...done.
> Loaded symbols for /lib/i686/cmov/libpthread.so.0
> Reading symbols from /lib/i686/cmov/libc.so.6...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/i686/cmov/libc.so.6
> Reading symbols from /lib/ld- linux.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/ld-linux.so.2
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/storage/posix.so...
> (no debugging symbols found)...done.
> Loaded symbols for
> /usr/lib/glusterfs/1.3.tla-322/xlator/storage/posix.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so...(no
> debugging
> symbols found)...done.
> Loaded symbols for
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/client.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so...(no debugging
> symbols found)...done.
> Loaded symbols for
> /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so...(no debugging
> symbols found)...done.
> Loaded symbols for
> /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io- threads.so...(no
> debugging symbols found)...done.
> Loaded symbols for
> /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so...
> (no debugging symbols found)...done.
> Loaded symbols for
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/transport/tcp/client.so...(no debugging
> symbols found)...done.
> Loaded symbols for
> /usr/lib/glusterfs/1.3.tla-322/transport/tcp/client.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/scheduler/rr.so...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/glusterfs/1.3.tla-322/scheduler/rr.so
> Reading symbols from
> /usr/lib/glusterfs/1.3.tla-322/transport/tcp/server.so...(no debugging
> symbols found)...done.
> Loaded symbols for
> /usr/lib/glusterfs/1.3.tla-322/transport/tcp/server.so
> Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/libgcc_s.so.1
>
> Core was generated by
> `[glusterfsd]
> '.
> Program terminated with signal 11, Segmentation fault.
> #0 0xb7ebc430 in pthread_mutex_lock () from
> /lib/i686/cmov/libpthread.so.0
> (gdb) backtrace
> #0 0xb7ebc430 in pthread_mutex_lock () from
> /lib/i686/cmov/libpthread.so.0
> #1 0xb7ed5989 in dict_ref () from /usr/lib/libglusterfs.so.0
> #2 0xb752b018 in fini () from
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
> #3 0xb752c185 in mop_lock () from
> /usr/lib/glusterfs/1.3.tla-322/xlator/protocol/server.so
> #4 0xb7538b1f in ?? () from
> /usr/lib/glusterfs/1.3.tla-322/xlator/performance/io-threads.so
> #5 0x081f1924 in ?? ()
> #6 0x081e2a08 in ?? ()
> #7 0x08051970 in ?? ()
> #8 0x00020000 in ?? ()
> #9 0x00000022 in ?? ()
> #10 0xbfadbdf0 in ?? ()
> #11 0xbfadbd48 in ?? ()
> #12 0xb753d1d5 in ?? () from
> /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/unify.so
> #13 0x081e2a08 in ?? ()
> #14 0x081ec2c8 in ?? ()
> #15 0x080518f8 in ?? ()
> #16 0x00020000 in ?? ()
> #17 0x00000022 in ?? ()
> #18 0xbfadbdf0 in ?? ()
> #19 0xbfadbd88 in ?? ()
> #20 0xb754bef3 in fini () from
> /usr/lib/glusterfs/1.3.tla-322/xlator/cluster/afr.so
> Previous frame inner to this frame (corrupt stack?)
>
> Thanks and regards
> Urban
>
>
>
>
>
> --
> Anand V. Avati
More information about the Gluster-devel
mailing list