[Gluster-devel] GlusterFS 1.3 mainlin 2.5-patch-317 crashes when second afr-node goes down

[Krishna Srinivas]

Hi Urban,

See if you find a core file in / of server1 and do
gdb glusterfsd -c <corefile>
and then type "backtrace" in gdb prompt.
This will give you the backtrace, can you mail it?

Thanks
Krishna

On 7/18/07, Urban Loesch <ul at enas.net> wrote:
> Hi,
>
> I checked out the latest source and compiled it with debian testing.
> I tried the setup with 2 Servers as described on
> http://www.gluster.org/docs/index.php/GlusterFS_High_Availability_Storage_with_GlusterFS
>
> (thanks to Paul England).
>
> Attached you can find server and client configuration.
> - I use 3 Servers with debian testing installed
> - fuse version:
> ii  fuse-utils               2.6.5-1                         Filesystem
> in USErspace (utilities)
> ii  libfuse-dev              2.6.5-1                         Filesystem
> in USErspace (development files)
> ii  libfuse2                 2.6.5-1                         Filesystem
> in USErspace library
> fuse init (API version 7.8)
> - 2 servers for Storage
> - 1 server as a client
>
> Everything works until I shutdown the second storage server, and try to
> write a file or do some ls on the client the "glusterfd" on server one
> will crash and the client gives me the error message:
>
> ls: /mnt/gluster/data1/: Transport endpoint is not connected
> ls: /mnt/gluster/data1/: Transport endpoint is not connected
>
> The log from server 1 is attached (error-server1.txt)
>
> Have you any idea where the error could be?
> If you need further information please let me know.
>
> Thanks and regards
> Urban Loesch
>
> 2007-07-18 16:50:14 E [tcp-client.c:170:tcp_connect] data1-gluster2-ds: non-blocking connect() returned: 111 (Connection refused)
> 2007-07-18 16:50:14 W [client-protocol.c:340:client_protocol_xfer] data1-gluster2-ds: not connected at the moment to submit frame type(0) op(22)
> 2007-07-18 16:50:14 D [tcp-client.c:70:tcp_connect] data1-gluster2-ds: socket fd = 3
> 2007-07-18 16:50:14 D [tcp-client.c:88:tcp_connect] data1-gluster2-ds: finalized on port `1023'
> 2007-07-18 16:50:14 D [tcp-client.c:109:tcp_connect] data1-gluster2-ds: defaulting remote-port to 6996
> 2007-07-18 16:50:14 D [tcp-client.c:141:tcp_connect] data1-gluster2-ds: connect on 3 in progress (non-blocking)
> 2007-07-18 16:50:14 E [tcp-client.c:170:tcp_connect] data1-gluster2-ds: non-blocking connect() returned: 111 (Connection refused)
> 2007-07-18 16:50:14 W [client-protocol.c:340:client_protocol_xfer] data1-gluster2-ds: not connected at the moment to submit frame type(0) op(20)
> 2007-07-18 16:50:14 E [afr.c:576:afr_getxattr_cbk] data1-ds-afr: (path=) op_ret=-1 op_errno=107
> 2007-07-18 16:50:14 C [common-utils.c:208:gf_print_trace] debug-backtrace: Got signal (11), printing backtrace
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(gf_print_trace+0x2b) [0xb7fc44fb]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /lib/i686/cmov/libc.so.6 [0xb7e7e208]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/protocol/server.so [0xb7607e1f]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0 [0xb7fc32a7]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/cluster/unify.so [0xb762410c]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/cluster/afr.so [0xb762cf4c]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/protocol/client.so [0xb7645da2]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/protocol/client.so [0xb763f160]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/protocol/client.so [0xb7640c9c]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/cluster/afr.so [0xb762d0da]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/cluster/unify.so(unify_getxattr+0x140) [0xb7624253]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(default_getxattr+0xe1) [0xb7fc338f]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/protocol/server.so [0xb760ce6f]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0 [0xb7fcc151]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(call_resume+0x33) [0xb7fce6b5]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/protocol/server.so [0xb760d067]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/protocol/server.so [0xb76114f1]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.pre6/xlator/protocol/server.so(notify+0xc9) [0xb7611e8b]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(transport_notify+0x62) [0xb7fc631f]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0 [0xb7fc6a28]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(sys_epoll_iteration+0x147) [0xb7fc6d0c]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(poll_iteration+0x1d) [0xb7fc654c]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: [glusterfsd] [0x8049340]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: /lib/i686/cmov/libc.so.6(__libc_start_main+0xe0) [0xb7e6a030]
> 2007-07-18 16:50:14 C [common-utils.c:210:gf_print_trace] debug-backtrace: [glusterfsd] [0x8048d51]
>
>    ### Add client feature and attach to remote subvolume
>    volume gluster1
>      type protocol/client