[Gluster-devel] Coredump generated by bug-808400-stripe.t (lock migration?)

[Niels de Vos]

A core file was generated for a regression test that is completely
unrelated to stripe or lock migration:

    http://build.gluster.org/job/rackspace-regression-2GB-triggered/9218/consoleFull

    [18:38:39] ./tests/bugs/protocol/bug-808400-stripe.t .. 
    not ok 9 
    not ok 10 
    not ok 11 
    Failed 3/13 subtests 
    [18:38:39]
    
    Test Summary Report
    -------------------
    ./tests/bugs/protocol/bug-808400-stripe.t (Wstat: 0 Tests: 13 Failed: 3)
      Failed tests:  9-11
    Files=1, Tests=13, 19 wallclock secs ( 0.02 usr  0.01 sys +  0.80 cusr  0.81 csys =  1.64 CPU)
    Result: FAIL
    ./tests/bugs/protocol/bug-808400-stripe.t: bad status 1
    ./tests/bugs/protocol/bug-808400-stripe.t: 1 new core files


These are the steps to download and extract the core and use the right
sources for debugging:

    $ cd /srv/src
    $ git clone http://review.gluster.org/glusterfs.git
    $ cd glusterfs
    $ git fetch origin refs/changes/03/10803/2
    $ git checkout -b core-9218 FETCH_HEAD

    $ cd /var/tmo
    $ wget http://slave25.cloud.gluster.org/archived_builds/build-install-20150517:16:40:01.tar.bz2
    $ mkdir core-9218
    $ cd core-9218
    $ tar xj < ../build-install-20150517\:16\:40\:01.tar.bz2
    $ gdb -ex 'set sysroot ./' \
        -ex 'core-file build/install/cores/core.13817' \
        build/install/sbin/glusterfsd

    (gdb) set substitute-path /home/jenkins/root/workspace/rackspace-regression-2GB-triggered /srv/src/glusterfs
    (gdb) bt
    #0  0x00007f28a7171116 in stripe_free_xattr_str (local=0x7f288c00377c)
        at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/stripe/src/stripe-helpers.c:152
    #1  0x00007f28a716eb11 in stripe_vgetxattr_cbk (frame=0x7f288c00367c, cookie=0x1, this=0x7f28a000aae0, op_ret=0, op_errno=0, dict=0x7f28a00349bc, xdata=0x0)
        at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/stripe/src/stripe.c:5451
    #2  0x00007f28a7395e3e in client3_3_fgetxattr_cbk (req=0x7f288c005ebc, iov=0x7f288c005efc, count=1, myframe=0x7f288c00570c)
        at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/protocol/client/src/client-rpc-fops.c:1165
    #3  0x00007f28b4027e08 in rpc_clnt_handle_reply (clnt=0x7f28a002e940, pollin=0x7f28a0034f80)
        at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:766
    #4  0x00007f28b4028228 in rpc_clnt_notify (trans=0x7f28a002edc0, mydata=0x7f28a002e970, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f28a0034f80)
        at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:894
    #5  0x00007f28b40247d4 in rpc_transport_notify (this=0x7f28a002edc0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f28a0034f80)
        at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-transport.c:543
    #6  0x00007f28a95e62ed in socket_event_poll_in (this=0x7f28a002edc0)
        at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2290
    #7  0x00007f28a95e67a8 in socket_event_handler (fd=10, idx=1, data=0x7f28a002edc0, poll_in=1, poll_out=0, poll_err=0)
        at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2403
    #8  0x00007f28b42d3041 in event_dispatch_epoll_handler (event_pool=0x1b3fc10, event=0x7f28a7fbfe70)
        at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:572
    #9  0x00007f28b42d339a in event_dispatch_epoll_worker (data=0x1b7e310)
        at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:674
    #10 0x00007f28b353f9d1 in start_thread () from ./lib64/libpthread.so.0
    #11 0x00007f28b2ea98fd in clone () from ./lib64/libc.so.6
    (gdb) f 0
    #0  0x00007f28a7171116 in stripe_free_xattr_str (local=0x7f288c00377c)
        at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/stripe/src/stripe-helpers.c:152
    152	                if (xattr && xattr->xattr_value)
    (gdb) l
    147	                goto out;
    148	
    149	        for (i = 0; i < local->nallocs; i++) {
    150	                xattr = local->xattr_list + i;
    151	
    152	                if (xattr && xattr->xattr_value)
    153	                        GF_FREE (xattr->xattr_value);
    154	        }
    155	
    156	        ret = 0;
    (gdb) p *local
    Attempt to resolve a variably-sized type which appears in the interior of a structure type
    (gdb) p *xattr
    Cannot access memory at address 0xadc0de00007f288c


This indicates a use-after-free problem. The 0xadc0de from *xattr would
be "0xdeadc0de", just a little chopped of. This pattern is written to
the area after GF_FREE() is called on a pointer when compiles with
--enable-debug.

The question now is, why got this free'd, and if that is correct, why is
the stripe xlator trying to free the xattr again?

At least, the above is my guessing. I can be wrong of course :-)

Who wants to look into this?

Thanks,
Niels