[Gluster-devel] crash in dht in https://build.gluster.org/job/rackspace-regression-2GB-triggered/18134/consoleFull

[Raghavendra Gowdappa]

Seems like a double unwind (or frame corruption in tier)
(gdb) down
#5  0x00007f3f8c34c0ac in dht_lookup_dir_cbk (frame=0x7f3f6c002a0c, cookie=0x7f3f6c0064bc, this=0x7f3f8803bad0, op_ret=-1, op_errno=2, inode=0x0, 
    stbuf=0x7f3f6c0069d4, xattr=0x0, postparent=0x7f3f6c006c04)
    at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:724
724	        ret = dht_selfheal_directory (frame, dht_lookup_selfheal_cbk,
(gdb) p *frame
$13 = {root = 0x29f00000000000, parent = 0xadc0de00007f3f6c, frames = {next = 0x50e4000000000de, prev = 0x3000007f3f88}, local = 0xac0000000000, 
  this = 0x29f00000000000, ret = 0xadc0de00007f3f6c, ref_count = 222, lock = 84819968, cookie = 0x3000007f3f88, complete = _gf_false, op = 44032, begin = {
    tv_sec = 11804356835803136, tv_usec = -5926493018029867156}, end = {tv_sec = 364298988607766750, tv_usec = 52776566472584}, 
  wind_from = 0xac0000000000 <Address 0xac0000000000 out of bounds>, wind_to = 0x29f00000000000 <Address 0x29f00000000000 out of bounds>, 
  unwind_from = 0xffffff00007f3f6c <Address 0xffffff00007f3f6c out of bounds>, unwind_to = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>}
(gdb) p this->name
$14 = 0x7f3f8803b640 "patchy-tier-dht"

No, I am not aware of this crash.

----- Original Message -----
> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> To: "Gluster Devel" <gluster-devel at gluster.org>, "Raghavendra Gowdappa" <rgowdapp at redhat.com>, "Nithya Balachandran"
> <nbalacha at redhat.com>
> Sent: Wednesday, February 10, 2016 5:51:32 AM
> Subject: crash in dht in https://build.gluster.org/job/rackspace-regression-2GB-triggered/18134/consoleFull
> 
> hi,
>         I see the following crash. Is this a known issue?
> (gdb) bt
> #0  0x00007f3f8c339fb4 in dht_selfheal_dir_setattr
> (frame=0x7f3f6c002a0c, loc=0x7f3f6c000944, stbuf=0x7f3f6c0009d4,
> valid=16777215,
>      layout=0x7f3f6c004140) at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1087
> #1  0x00007f3f8c33a4f8 in dht_selfheal_dir_mkdir_cbk
> (frame=0x7f3f6c002a0c, cookie=0x7f3f9004201c, this=0x7f3f8803bad0,
> op_ret=-1, op_errno=5,
>      inode=0x0, stbuf=0x0, preparent=0x0, postparent=0x0, xdata=0x0)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1147
> #2  0x00007f3f8fb81f31 in dht_mkdir (frame=0x7f3f9004201c,
> this=0x7f3f8803aa80, loc=0x7f3f6c000944, mode=16877, umask=0,
> params=0x7f3f9003594c)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:6710
> #3  0x00007f3f8c33ad1f in dht_selfheal_dir_mkdir (frame=0x7f3f6c002a0c,
> loc=0x7f3f6c000944, layout=0x7f3f6c004140, force=0)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1256
> #4  0x00007f3f8c33c4c8 in dht_selfheal_directory (frame=0x7f3f6c002a0c,
> dir_cbk=0x7f3f8c349588 <dht_lookup_selfheal_cbk>, loc=0x7f3f6c000944,
>      layout=0x7f3f6c004140) at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1837
> #5  0x00007f3f8c34c0ac in dht_lookup_dir_cbk (frame=0x7f3f6c002a0c,
> cookie=0x7f3f6c0064bc, this=0x7f3f8803bad0, op_ret=-1, op_errno=2,
> inode=0x0,
>      stbuf=0x7f3f6c0069d4, xattr=0x0, postparent=0x7f3f6c006c04)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:724
> #6  0x00007f3f8fb5ea98 in dht_lookup_dir_cbk (frame=0x7f3f6c0064bc,
> cookie=0x7f3f6c0084dc, this=0x7f3f8803aa80, op_ret=-1, op_errno=2,
>      inode=0x7f3f6c0013ec, stbuf=0x7f3f94bc2860, xattr=0x0,
> postparent=0x7f3f94bc27f0)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:714
> #7  0x00007f3f8fddb23a in client3_3_lookup_cbk (req=0x7f3f6c0090ac,
> iov=0x7f3f6c0090ec, count=1, myframe=0x7f3f6c0084dc)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/protocol/client/src/client-rpc-fops.c:3028
> #8  0x00007f3fa0c2d42a in rpc_clnt_handle_reply (clnt=0x7f3f8806cdb0,
> pollin=0x7f3f90041c90)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:759
> #9  0x00007f3fa0c2d8c8 in rpc_clnt_notify (trans=0x7f3f8806d240,
> mydata=0x7f3f8806cde0, event=RPC_TRANSPORT_MSG_RECEIVED,
> data=0x7f3f90041c90)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:900
> #10 0x00007f3fa0c29b5a in rpc_transport_notify (this=0x7f3f8806d240,
> event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f3f90041c90)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-transport.c:541
> #11 0x00007f3f961eadcb in socket_event_poll_in (this=0x7f3f8806d240)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2231
> #12 0x00007f3f961eb321 in socket_event_handler (fd=18, idx=12,
> data=0x7f3f8806d240, poll_in=1, poll_out=0, poll_err=0)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2344
> #13 0x00007f3fa0ec61a8 in event_dispatch_epoll_handler
> (event_pool=0x121fce0, event=0x7f3f94bc2e70)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:571
> #14 0x00007f3fa0ec6596 in event_dispatch_epoll_worker (data=0x125e1a0)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:674
> #15 0x00007f3fa0144a51 in start_thread () from ./lib64/libpthread.so.0
> #16 0x00007f3f9faae93d in clone () from ./lib64/libc.so.6
> 
> 
> Pranith
>