[Gluster-devel] Re: more bugs (was Re: io-threads...)

[Brent A Nelson]

Wow, it almost looked like the patch fixed the issue with using 
stat-prefetch, but see below. I was almost unable to get it to crash with 
du's or rm's on complex directories, as it did fairly easily before.

Also, I think it fixed a tiny anomaly that I had noticed but ignored. 
Previously, even without stat-prefetch, multiple du's on a complex 
directory could give slightly different total sizes (a few KB out of 
many GB).  Now, there is no such fluctuation at all.

I WAS able to get a crash with one machine doing du, while a different 
machine did removes of the same area.  The du machine is the one where the 
glusterfs client died (the first du completed, the second died). The 
glusterfs left a backtrace in the log but no core, perhaps because I 
compiled with CFLAGS=-O3.  See attached backtrace.

Stat-prefetch did at least withstand a great deal more torture than before 
the patch, so it seems to be a significant improvement.  Note that I 
haven't tried the new patch without stat-prefetch, so it's possible that 
heavy testing might be able to kill it even without stat-prefetch; I'm not 
sure.

Thanks,

Brent

PS Alas, there was no effect on the NFS reexport issue.
PPS The AFR client failover works pretty well, but I notice something. 
The first attempt to access the glusterfs after losing contact with a 
glusterfsd is sometimes faulty (e.g., the first df may say it's not 
connected or give a smaller size for the volume; trying to cat a file may 
not work on the first try).  The very next attempt will succeed, however.

On Tue, 1 May 2007, Anand Avati wrote:

>
>> I was wondering if you could describe patch-134 a little? I was curious as
>> to whether or not it could be related to the stat-prefetch or the NFS
>> reexport issues.
>
> this was a bug in afr which could have triggered for anybody who used
> AFR and accessed a directory. the functions forming the reply path of
> a transaction use function pointers and the afr's opendir reply
> callback prototype had an extra member and derefered that pointer
> (which is a junk pointer). so far all of us were lucky that the
> derefernced pointer happened to point to some allocated memory (though
> nothing was altered or used). it is very much possible that this culd
> be related to the stat-prefetch. the latest glusterfs codebase now
> prints a backtrace of a segfaul in the log as well as dumps a core,
> next time if you get a segfault please pass on the core and/or log.
> I do not see how nfs rexport can be affected, but you never know if
> this could have triggered a side effect somewhere else.
>
> I have done only a very little check with NFS re-export. once 1.3 next
> release is done i will do a more thorough check.
>
> regards,
> avati
>
> -- 
> ultimate_answer_t
> deep_thought (void)
> {
>  sleep (years2secs (7500000));
>  return 42;
> }
>
-------------- next part --------------
[May 01 15:48:48] [CRITICAL/common-utils.c:215/gf_print_trace()] debug-backtrace:Got signal (11), printing backtrace
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/usr/lib/libglusterfs.so.0(gf_print_trace+0x2d) [0xb7f04c8d]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:[0xffffe420]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/usr/lib/libglusterfs.so.0 [0xb7f02e19]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/usr/lib/libglusterfs.so.0 [0xb7f02e19]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/usr/lib/glusterfs/1.3.0-pre3/xlator/performance/stat-prefetch.so [0xb7f0ddd4]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/usr/lib/glusterfs/1.3.0-pre3/xlator/cluster/unify.so [0xb7549371]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/usr/lib/glusterfs/1.3.0-pre3/xlator/cluster/afr.so [0xb754f398]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/usr/lib/glusterfs/1.3.0-pre3/xlator/protocol/client.so [0xb755a637]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/usr/lib/glusterfs/1.3.0-pre3/xlator/protocol/client.so [0xb75597d2]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/usr/lib/libglusterfs.so.0(transport_notify+0x1d) [0xb7f0616d]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/usr/lib/libglusterfs.so.0(sys_epoll_iteration+0xe7) [0xb7f06e17]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/usr/lib/libglusterfs.so.0(poll_iteration+0x1d) [0xb7f0621d]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:[glusterfs] [0x804a8ce]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:/lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xdc) [0xb7da1ebc]
[May 01 15:48:48] [CRITICAL/common-utils.c:217/gf_print_trace()] debug-backtrace:[glusterfs] [0x804a071]