[Gluster-devel] crash in dht in https://build.gluster.org/job/rackspace-regression-2GB-triggered/18134/consoleFull
Raghavendra Gowdappa
rgowdapp at redhat.com
Wed Feb 10 04:51:46 UTC 2016
----- Original Message -----
> From: "Raghavendra Gowdappa" <rgowdapp at redhat.com>
> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Nithya Balachandran" <nbalacha at redhat.com>
> Sent: Wednesday, February 10, 2016 10:16:45 AM
> Subject: Re: crash in dht in https://build.gluster.org/job/rackspace-regression-2GB-triggered/18134/consoleFull
>
> Seems like a double unwind (or frame corruption in tier)
> (gdb) down
> #5 0x00007f3f8c34c0ac in dht_lookup_dir_cbk (frame=0x7f3f6c002a0c,
> cookie=0x7f3f6c0064bc, this=0x7f3f8803bad0, op_ret=-1, op_errno=2,
> inode=0x0,
> stbuf=0x7f3f6c0069d4, xattr=0x0, postparent=0x7f3f6c006c04)
> at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:724
> 724 ret = dht_selfheal_directory (frame, dht_lookup_selfheal_cbk,
> (gdb) p *frame
> $13 = {root = 0x29f00000000000, parent = 0xadc0de00007f3f6c, frames = {next =
> 0x50e4000000000de, prev = 0x3000007f3f88}, local = 0xac0000000000,
> this = 0x29f00000000000, ret = 0xadc0de00007f3f6c, ref_count = 222, lock =
> 84819968, cookie = 0x3000007f3f88, complete = _gf_false, op = 44032, begin
> = {
> tv_sec = 11804356835803136, tv_usec = -5926493018029867156}, end =
> {tv_sec = 364298988607766750, tv_usec = 52776566472584},
> wind_from = 0xac0000000000 <Address 0xac0000000000 out of bounds>, wind_to
> = 0x29f00000000000 <Address 0x29f00000000000 out of bounds>,
> unwind_from = 0xffffff00007f3f6c <Address 0xffffff00007f3f6c out of
> bounds>, unwind_to = 0xffffffffffffffff <Address 0xffffffffffffffff out of
> bounds>}
> (gdb) p this->name
> $14 = 0x7f3f8803b640 "patchy-tier-dht"
(gdb) up
#6 0x00007f3f8fb5ea98 in dht_lookup_dir_cbk (frame=0x7f3f6c0064bc, cookie=0x7f3f6c0084dc, this=0x7f3f8803aa80, op_ret=-1, op_errno=2, inode=0x7f3f6c0013ec,
stbuf=0x7f3f94bc2860, xattr=0x0, postparent=0x7f3f94bc27f0)
at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:714
714 DHT_STACK_UNWIND (lookup, frame, local->op_ret, local->op_errno,
(gdb) p this->name
$22 = 0x7f3f88039510 "patchy-hot-dht"
(gdb) p frame->parent
$23 = (call_frame_t *) 0xadc0de00007f3f6c
(gdb) p *frame->parent
Cannot access memory at address 0xadc0de00007f3f6c
(gdb) p *frame
$24 = {root = 0x64a00000000000, parent = 0xadc0de00007f3f6c, frames = {next = 0x50e4000000000de, prev = 0x3000007f3f88}, local = 0xac0000000000,
this = 0x64a00000000000, ret = 0xadc0de00007f3f6c, ref_count = 222, lock = 84819968, cookie = 0x3000007f3f88, complete = _gf_false, op = 44032, begin = {
tv_sec = 28323419531509760, tv_usec = -5926493018029867156}, end = {tv_sec = 364298988607766750, tv_usec = 52776566472584},
wind_from = 0xac0000000000 <Address 0xac0000000000 out of bounds>, wind_to = 0x64a00000000000 <Address 0x64a00000000000 out of bounds>,
unwind_from = 0xffffff00007f3f6c <Address 0xffffff00007f3f6c out of bounds>, unwind_to = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>}
(gdb) p this->name
$25 = 0x7f3f88039510 "patchy-hot-dht"
(gdb) up
#7 0x00007f3f8fddb23a in client3_3_lookup_cbk (req=0x7f3f6c0090ac, iov=0x7f3f6c0090ec, count=1, myframe=0x7f3f6c0084dc)
at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/protocol/client/src/client-rpc-fops.c:3028
3028 /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/protocol/client/src/client-rpc-fops.c: No such file or directory.
(gdb) p frame
$26 = (call_frame_t *) 0x7f3f6c0084dc
(gdb) p *frame
$27 = {root = 0x84c00000000000, parent = 0xadc0de00007f3f6c, frames = {next = 0x54d2000000000de, prev = 0x3000007f3f88}, local = 0xac0000000000,
this = 0x84c00000000000, ret = 0xadc0de00007f3f6c, ref_count = 222, lock = 88940544, cookie = 0x3000007f3f88, complete = _gf_false, op = 44032, begin = {
tv_sec = 37365803158339584, tv_usec = -5926493018029867156}, end = {tv_sec = 381996727768449246, tv_usec = 52776566472584},
wind_from = 0xac0000000000 <Address 0xac0000000000 out of bounds>, wind_to = 0x84c00000000000 <Address 0x84c00000000000 out of bounds>,
unwind_from = 0xffffff00007f3f6c <Address 0xffffff00007f3f6c out of bounds>, unwind_to = 0xffffffffffffffff <Address 0xffffffffffffffff out of bounds>}
(gdb) p this->name
$28 = 0x7f3f880202d0 "patchy-client-5"
Seems like entire stack is corrupted
> No, I am not aware of this crash.
>
> ----- Original Message -----
> > From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> > To: "Gluster Devel" <gluster-devel at gluster.org>, "Raghavendra Gowdappa"
> > <rgowdapp at redhat.com>, "Nithya Balachandran"
> > <nbalacha at redhat.com>
> > Sent: Wednesday, February 10, 2016 5:51:32 AM
> > Subject: crash in dht in
> > https://build.gluster.org/job/rackspace-regression-2GB-triggered/18134/consoleFull
> >
> > hi,
> > I see the following crash. Is this a known issue?
> > (gdb) bt
> > #0 0x00007f3f8c339fb4 in dht_selfheal_dir_setattr
> > (frame=0x7f3f6c002a0c, loc=0x7f3f6c000944, stbuf=0x7f3f6c0009d4,
> > valid=16777215,
> > layout=0x7f3f6c004140) at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1087
> > #1 0x00007f3f8c33a4f8 in dht_selfheal_dir_mkdir_cbk
> > (frame=0x7f3f6c002a0c, cookie=0x7f3f9004201c, this=0x7f3f8803bad0,
> > op_ret=-1, op_errno=5,
> > inode=0x0, stbuf=0x0, preparent=0x0, postparent=0x0, xdata=0x0)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1147
> > #2 0x00007f3f8fb81f31 in dht_mkdir (frame=0x7f3f9004201c,
> > this=0x7f3f8803aa80, loc=0x7f3f6c000944, mode=16877, umask=0,
> > params=0x7f3f9003594c)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:6710
> > #3 0x00007f3f8c33ad1f in dht_selfheal_dir_mkdir (frame=0x7f3f6c002a0c,
> > loc=0x7f3f6c000944, layout=0x7f3f6c004140, force=0)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1256
> > #4 0x00007f3f8c33c4c8 in dht_selfheal_directory (frame=0x7f3f6c002a0c,
> > dir_cbk=0x7f3f8c349588 <dht_lookup_selfheal_cbk>, loc=0x7f3f6c000944,
> > layout=0x7f3f6c004140) at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-selfheal.c:1837
> > #5 0x00007f3f8c34c0ac in dht_lookup_dir_cbk (frame=0x7f3f6c002a0c,
> > cookie=0x7f3f6c0064bc, this=0x7f3f8803bad0, op_ret=-1, op_errno=2,
> > inode=0x0,
> > stbuf=0x7f3f6c0069d4, xattr=0x0, postparent=0x7f3f6c006c04)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:724
> > #6 0x00007f3f8fb5ea98 in dht_lookup_dir_cbk (frame=0x7f3f6c0064bc,
> > cookie=0x7f3f6c0084dc, this=0x7f3f8803aa80, op_ret=-1, op_errno=2,
> > inode=0x7f3f6c0013ec, stbuf=0x7f3f94bc2860, xattr=0x0,
> > postparent=0x7f3f94bc27f0)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/cluster/dht/src/dht-common.c:714
> > #7 0x00007f3f8fddb23a in client3_3_lookup_cbk (req=0x7f3f6c0090ac,
> > iov=0x7f3f6c0090ec, count=1, myframe=0x7f3f6c0084dc)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/protocol/client/src/client-rpc-fops.c:3028
> > #8 0x00007f3fa0c2d42a in rpc_clnt_handle_reply (clnt=0x7f3f8806cdb0,
> > pollin=0x7f3f90041c90)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:759
> > #9 0x00007f3fa0c2d8c8 in rpc_clnt_notify (trans=0x7f3f8806d240,
> > mydata=0x7f3f8806cde0, event=RPC_TRANSPORT_MSG_RECEIVED,
> > data=0x7f3f90041c90)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-clnt.c:900
> > #10 0x00007f3fa0c29b5a in rpc_transport_notify (this=0x7f3f8806d240,
> > event=RPC_TRANSPORT_MSG_RECEIVED, data=0x7f3f90041c90)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-transport.c:541
> > #11 0x00007f3f961eadcb in socket_event_poll_in (this=0x7f3f8806d240)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2231
> > #12 0x00007f3f961eb321 in socket_event_handler (fd=18, idx=12,
> > data=0x7f3f8806d240, poll_in=1, poll_out=0, poll_err=0)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2344
> > #13 0x00007f3fa0ec61a8 in event_dispatch_epoll_handler
> > (event_pool=0x121fce0, event=0x7f3f94bc2e70)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:571
> > #14 0x00007f3fa0ec6596 in event_dispatch_epoll_worker (data=0x125e1a0)
> > at
> > /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:674
> > #15 0x00007f3fa0144a51 in start_thread () from ./lib64/libpthread.so.0
> > #16 0x00007f3f9faae93d in clone () from ./lib64/libc.so.6
> >
> >
> > Pranith
> >
>
More information about the Gluster-devel
mailing list