[Gluster-devel] Re: client cannot maintain mount of unified AFR

Thu Jun 28 20:46:02 UTC 2007

Majied,

This looks like the same bug in io-threads that I have already reported. Try turning off io-threads and see if your problem goes away. 

Harris

----- Original Message -----
From: "Majied Najjar" <majied.najjar at nationalnet.com>
To: gluster-devel at nongnu.org
Sent: Thursday, June 28, 2007 3:07:20 PM (GMT-0500) America/New_York
Subject: Re: [Gluster-devel] Re: client cannot maintain mount of unified AFR

Here is the debug-backtrace from glusterfsd right during/after it dies:

2007-06-28 14:59:57 E [protocol.c:262:gf_block_unserialize_transport] libglusterfs/protocol: full_read of header failed: peer (127.0.0.1)
2007-06-28 14:59:57 C [tcp.c:81:tcp_disconnect] server: connection disconnected
2007-06-28 14:59:57 E [protocol.c:262:gf_block_unserialize_transport] libglusterfs/protocol: full_read of header failed: peer (127.0.0.1)
2007-06-28 14:59:57 C [tcp.c:81:tcp_disconnect] server: connection disconnected
2007-06-28 14:59:57 E [protocol.c:262:gf_block_unserialize_transport] libglusterfs/protocol: full_read of header failed: peer (127.0.0.1)
2007-06-28 14:59:57 C [tcp.c:81:tcp_disconnect] server: connection disconnected
2007-06-28 14:59:57 E [protocol.c:262:gf_block_unserialize_transport] libglusterfs/protocol: full_read of header failed: peer (127.0.0.1)
2007-06-28 14:59:57 C [tcp.c:81:tcp_disconnect] server: connection disconnected
2007-06-28 14:59:57 E [protocol.c:262:gf_block_unserialize_transport] libglusterfs/protocol: full_read of header failed: peer (127.0.0.1)
2007-06-28 14:59:57 C [tcp.c:81:tcp_disconnect] server: connection disconnected
2007-06-28 15:00:47 C [common-utils.c:205:gf_print_trace] debug-backtrace: Got signal (11), printing backtrace
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(gf_print_trace+0x2e) [0xb7f2c54e]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: [0xffffe420]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(dict_destroy+0x4e) [0xb7f2626e]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(dict_unref+0x4e) [0xb7f2630e]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0 [0xb7f302d0]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(call_resume+0x67) [0xb7f30417]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.0-pre5/xlator/performance/io-threads.so [0xb75a01ef]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libpthread.so.0 [0xb7ef4240]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6(__clone+0x5e) [0xb7e893de]
2007-06-28 15:00:47 C [common-utils.c:205:gf_print_trace] debug-backtrace: Got signal (11), printing backtrace
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(gf_print_trace+0x2e) [0xb7fbf54e]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: [0xffffe420]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(dict_destroy+0x4e) [0xb7fb926e]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(dict_unref+0x4e) [0xb7fb930e]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0 [0xb7fc32d0]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(call_resume+0x67) [0xb7fc3417]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.0-pre5/xlator/performance/io-threads.so [0xb76331ef]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libpthread.so.0 [0xb7f87240]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6(__clone+0x5e) [0xb7f1c3de]
2007-06-28 15:00:47 C [common-utils.c:205:gf_print_trace] debug-backtrace: Got signal (6), printing backtrace
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(gf_print_trace+0x2e) [0xb7eda54e]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: [0xffffe420]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6(abort+0x109) [0xb7d95fb9]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6 [0xb7dc9d3a]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6 [0xb7dcfd8c]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6 [0xb7dd139f]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6(__libc_free+0x82) [0xb7dd1672]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(call_resume+0x40) [0xb7ede3f0]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.0-pre5/xlator/performance/io-threads.so [0xb754e1ef]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libpthread.so.0 [0xb7ea2240]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6(__clone+0x5e) [0xb7e373de]
2007-06-28 15:00:47 C [common-utils.c:205:gf_print_trace] debug-backtrace: Got signal (6), printing backtrace
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(gf_print_trace+0x2e) [0xb7ef054e]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: [0xffffe420]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6(abort+0x109) [0xb7dabfb9]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6 [0xb7ddfd3a]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6 [0xb7de5d8c]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6 [0xb7de739f]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6(__libc_free+0x82) [0xb7de7672]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/libglusterfs.so.0(call_resume+0x40) [0xb7ef43f0]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /usr/lib/glusterfs/1.3.0-pre5/xlator/performance/io-threads.so [0xb75641ef]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libpthread.so.0 [0xb7eb8240]
2007-06-28 15:00:47 C [common-utils.c:207:gf_print_trace] debug-backtrace: /lib/tls/i686/cmov/libc.so.6(__clone+0x5e) [0xb7e4d3de]

Majied

On Thu, 28 Jun 2007 11:32:53 -0400
Majied Najjar <majied.najjar at nationalnet.com> wrote:

> Yes. After I sent this message, I realized that I neglected to upgrade the client.  However, after I upgraded the client and updated the config to include the namespace info, the servers kept crashing.  Since this was a production machine, I had to downgrade as my "maintenance window" was over. :-)
> 
> In an effort to get more data, I set up another instance on a testing server and got somewhat similar results.  I have placed my core file from the server crash at http://majied.net/core.txt and my client/server config at http://majied.net/client-server.txt .
> 
> Let me know if you need more information.
> 
> Thanks,
> Majied
> 
> 
> On Thu, 28 Jun 2007 18:43:34 +0530
> "Anand Avati" <avati at zresearch.com> wrote:
> 
> > You would get a 'connection refused' if the server is not running. Can you
> > please check if glusterfsd was running at that moment? also please get the
> > logs of the glusterfsd which was not running (and if possible, the core
> > dump's backtrace)
> > 
> > Also have you upgraded all the servers and clients?
> > 
> > thanks,
> > avati
> > 
> > 
> > 2007/6/27, Majied Najjar <majied.najjar at nationalnet.com>:
> > >
> > > Also,
> > >
> > > This happened when the first client in the client config
> > > rebooted.  Normally, the second client in the afr group would have picked up
> > > the slack, but instead I was getting connection refused from the client.  I
> > > am assuming this is a locking issue?
> > >
> > > Majied Najjar
> > 
> > 
> > 
> > -- 
> > Anand V. Avati
> 
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel

_______________________________________________
Gluster-devel mailing list
Gluster-devel at nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel