[Gluster-devel] failure in tests/bugs/glusterd/bug-974007.t

[Nithya Balachandran]

Looks like glusterd crashed:

23:49:02] ./tests/bugs/glusterd/bug-974007.t .. 
not ok 9 Got "" instead of "completed"
not ok 10 Got "failed" instead of "success"
not ok 11 Got "" instead of "Replicate"
not ok 13 
not ok 14 
Failed 5/15 subtests 
[23:49:02]



The failed test 9 and 10 are:

# Wait for rebalance to complete
EXPECT_WITHIN $REBALANCE_TIMEOUT "completed" remove_brick_status_completed_field "$V0" "$H0:$B0/${V0}6 $H0:$B0/${V0}1 $H0:$B0/${V0}2 $H0:$B0/${V0}5"

# Check commit status
function remove_brick_commit_status {
        $CLI volume remove-brick $V0 \
         $H0:$B0/${V0}6  $H0:$B0/${V0}1 \
         $H0:$B0/${V0}2  $H0:$B0/${V0}5 commit 2>&1 |grep -oE "success|failed"
}
EXPECT "success" remove_brick_commit_status;





Core was generated by `glusterd'.
Program terminated with signal 6, Aborted.
#0  0x00007efff4778625 in raise () from ./lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install device-mapper-event-libs-1.02.90-2.el6.x86_64 device-mapper-libs-1.02.90-2.el6.x86_64 glibc-2.12-1.149.el6.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-33.el6.x86_64 libcom_err-1.41.12-21.el6.x86_64 libgcc-4.4.7-11.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 libsepol-2.0.41-4.el6.x86_64 libudev-147-2.57.el6.x86_64 libxml2-2.7.6-14.el6_5.2.x86_64 lvm2-libs-2.02.111-2.el6.x86_64 openssl-1.0.1e-30.el6.x86_64 userspace-rcu-0.7.7-1.el6.x86_64 zlib-1.2.3-29.el6.x86_64
(gdb) bt
#0  0x00007efff4778625 in raise () from ./lib64/libc.so.6
#1  0x00007efff4779e05 in abort () from ./lib64/libc.so.6
#2  0x00007efff47b6537 in __libc_message () from ./lib64/libc.so.6
#3  0x00007efff47bbe66 in malloc_printerr () from ./lib64/libc.so.6
#4  0x00007efff47be9b3 in _int_free () from ./lib64/libc.so.6
#5  0x00007efff59a0117 in __gf_free (free_ptr=0x7effe0083200)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/mem-pool.c:318
#6  0x00007efff59a0909 in mem_put (ptr=0x7effe008321c)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/mem-pool.c:570
#7  0x00007efff572abd0 in rpc_clnt_submit (rpc=0x7effe007a7e0, prog=0x7effeab0c9a0, procnum=6, cbkfn=0x7effea805538 <glusterd_brick_op_cbk>, 
    proghdr=0x7effdc4090b0, proghdrcount=1, progpayload=0x0, progpayloadcount=0, iobref=0x7effd01a3d60, frame=0x7efff37c9e00, rsphdr=0x0, 
    rsphdr_count=0, rsp_payload=0x0, rsp_payload_count=0, rsp_iobref=0x0)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:1603
#8  0x00007effea7e1ab2 in glusterd_submit_request_unlocked (rpc=0x7effe007a7e0, req=0x7effd019c960, frame=0x7efff37c9e00, prog=0x7effeab0c9a0, 
    procnum=6, iobref=0x7effd01a3d60, this=0x1b36cf0, cbkfn=0x7effea805538 <glusterd_brick_op_cbk>, 
    xdrproc=0x404c48 <xdr_gd1_mgmt_brick_op_req at plt>)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-utils.c:287
#9  0x00007effea7e1b81 in glusterd_submit_request (rpc=0x7effe007a7e0, req=0x7effd019c960, frame=0x7efff37c9e00, prog=0x7effeab0c9a0, 
    procnum=6, iobref=0x0, this=0x1b36cf0, cbkfn=0x7effea805538 <glusterd_brick_op_cbk>, xdrproc=0x404c48 <xdr_gd1_mgmt_brick_op_req at plt>)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-utils.c:319
#10 0x00007effea805a4e in glusterd_brick_op (frame=0x0, this=0x1b36cf0, data=0x7effd01a1100)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-rpc-ops.c:2164
#11 0x00007effea7dfb40 in glusterd_op_ac_send_brick_op (event=0x7effd01a91d0, ctx=0x0)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-op-sm.c:6335
#12 0x00007effea7e077b in glusterd_op_sm ()
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-op-sm.c:6901
#13 0x00007effea82c2fa in __glusterd_handle_defrag_volume (req=0x1b6f5ec)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-rebalance.c:513
#14 0x00007effea7be687 in glusterd_big_locked_handler (req=0x1b6f5ec, actor_fn=0x7effea82bf2b <__glusterd_handle_defrag_volume>)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:83
#15 0x00007effea82c39a in glusterd_handle_defrag_volume (req=0x1b6f5ec)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/mgmt/glusterd/src/glusterd-rebalance.c:531
#16 0x00007efff59b2b4a in synctask_wrap (old_task=0x7effdc00ae30)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/syncop.c:375
#17 0x00007efff47898f0 in ?? () from ./lib64/libc.so.6
#18 0x0000000000000000 in ?? ()
(gdb) 


Atin, can someone  from the glusterd team take a look at it?

Regards,
Nithya

----- Original Message -----
> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> To: "Nithya Balachandran" <nbalacha at redhat.com>, "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Thursday, May 7, 2015 3:36:31 PM
> Subject: failure in tests/bugs/glusterd/bug-974007.t
> 
> Nitya,
>        Seems like rebalance is not completing in this test? Could you
> take a look.
> http://build.gluster.org/job/rackspace-regression-2GB-triggered/8595/consoleFull
> 
> Pranith
>