[Gluster-devel] brick stop responding (3.4.0qa8)

[Emmanuel Dreyfus]

On Sun, Feb 03, 2013 at 02:13:40PM -0800, Anand Avati wrote:
> Yeah, a lot of threads are "missing"! Do the logs have anything unusual?

I understand the threads are here, but the process is corrupted enough
that gdb cannot figure them out. A kernel thread shows 15 threads operating
and they do not terminate before it stops responding.

Running with electric fence leads to an early crash. I am not sure
it is related, but it is probably worth a fix:

Program terminated with signal 11, Segmentation fault.
#0  slotForUserAddress (address=0x7f7ff2bafff4) at efence.c:648
648     efence.c: No such file or directory.
        in efence.c
(gdb) bt
#0  slotForUserAddress (address=0x7f7ff2bafff4) at efence.c:648
#1  free (address=0x7f7ff2bafff4) at efence.c:713
#2  0x00007f7ff744a31a in runner_end (runner=0x7f7ff1bfeef0) at run.c:370
#3  0x00007f7ff744ac9e in runner_run_generic (runner=0x7f7ff1bfeef0, 
    rfin=0x7f7ff744a2f6 <runner_end>) at run.c:386
#4  0x00007f7ff343ddb9 in glusterd_volume_start_glusterfs (
    volinfo=0x7f7ff370fae0, brickinfo=0x7f7ff37407a8, wait=_gf_true)
    at glusterd-utils.c:1337
#5  0x00007f7ff344366b in glusterd_brick_start (volinfo=0x7f7ff370fae0, 
    brickinfo=0x7f7ff37407a8, wait=_gf_true) at glusterd-utils.c:3961
#6  0x00007f7ff3447328 in glusterd_restart_bricks (conf=0x7f7ff73f2a98)
    at glusterd-utils.c:3991
#7  0x00007f7ff743c9c4 in synctask_wrap (old_task=<optimized out>)
    at syncop.c:129
#8  0x00007f7ff5e580a0 in swapcontext () from /usr/lib/libc.so.12

In runner_end():
368             if (runner->argv) {
369                     for (p = runner->argv; *p; p++)
370                             GF_FREE (*p);

Inspection of runner->argv shows it is not NULL terminated. electric-fence
with EF_PROTECT_BELOW  cause us to crash here:
0x7f7ff2ba7e00: 0xf2babfe4      0x00007f7f      0xf2badffc      0x00007f7f
0x7f7ff2ba7e10: 0xf2bafff4      0x00007f7f      0xf2bb1ff0      0x00007f7f
0x7f7ff2ba7e20: 0xf2bb3fe4      0x00007f7f      0xf2bb5ffc      0x00007f7f
0x7f7ff2ba7e30: 0xf2bb7fc4      0x00007f7f      0xf2bb9ffc      0x00007f7f
0x7f7ff2ba7e40: 0xf2bbbfcc      0x00007f7f      0xf2bbdff0      0x00007f7f
0x7f7ff2ba7e50: 0xf2bbfff0      0x00007f7f      0xf2bc1ffc      0x00007f7f
0x7f7ff2ba7e60: 0xf2bc3fcc      0x00007f7f      0xf2bc5ff0      0x00007f7f
0x7f7ff2ba7e70: 0xf2bc7fc4      0x00007f7f      0xf2bc9ff0      0x00007f7f
0x7f7ff2ba7e80: 0xf2bcbff8      0x00007f7f      0xf2bcdff0      0x00007f7f
0x7f7ff2ba7e90: 0xf2bcffe0      0x00007f7f      Cannot access memory at address 0x7f7ff2ba7e98

In case it helps, the last element:
(gdb) x/1s 0x00007f7ff2bcffe0
0x7f7ff2bcffe0:  "gfs33-server.listen-port=49152"

It is indeed the last one reported by ps:
PID TTY   STAT    TIME COMMAND
626 ?     Ssl  0:00.34 /usr/local/sbin/glusterfsd -s localhost --volfile-id gfs33.hotstuff.export-wd1a -p /var/lib/glusterd/vols/gfs33/run/hotstuff-export-wd1a.pid -S /var/run/66fedc0377e53ff9d6523d0802a230d1.socket --brick-name /export/wd1a -l /usr/local/var/log/glusterfs/bricks/export-wd1a.log --xlator-option *-posix.glusterd-uuid=2dbb8fc1-c2ab-4992-b080-fdc8556d1e34 --brick-port 49152 --xlator-option gfs33-server.listen-port=49152





-- 
Emmanuel Dreyfus
manu at netbsd.org