[Gluster-devel] USS test cases failure with core

[Avra Sengupta]

Hi,

I had a look at the bt:

(gdb) bt
#0  0x00007f8a40d4c175 in dict_unserialize (orig_buf=0x243e840 "", size=504, fill=0x7fffe8386aa8) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/dict.c:2692
#1  0x00007f8a2f580f73 in server_setvolume (req=0x7f8a2efeeb58) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/xlators/protocol/server/src/server-handshake.c:379
#2  0x00007f8a40b17da6 in rpcsvc_handle_rpc_call (svc=0x24159c0, trans=0x243d510, msg=0x2422010) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpcsvc.c:682
#3  0x00007f8a40b18119 in rpcsvc_notify (trans=0x243d510, mydata=0x24159c0, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x2422010)
     at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpcsvc.c:776
#4  0x00007f8a40b1d2cc in rpc_transport_notify (this=0x243d510, event=RPC_TRANSPORT_MSG_RECEIVED, data=0x2422010)
     at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-lib/src/rpc-transport.c:516
#5  0x00007f8a35d01fdf in socket_event_poll_in (this=0x243d510) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2233
#6  0x00007f8a35d0249a in socket_event_handler (fd=10, idx=5, data=0x243d510, poll_in=1, poll_out=0, poll_err=0)
     at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/rpc/rpc-transport/socket/src/socket.c:2346
#7  0x00007f8a40dc0d8f in event_dispatch_epoll_handler (event_pool=0x23db3c0, events=0x23f9ff0, i=0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:384
#8  0x00007f8a40dc0f89 in event_dispatch_epoll (event_pool=0x23db3c0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event-epoll.c:445
#9  0x00007f8a40d8dd93 in event_dispatch (event_pool=0x23db3c0) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/event.c:113
#10 0x00000000004097b6 in main (argc=17, argv=0x7fffe83880c8) at /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd.c:2046
(gdb)

The crash is at dict_unserialize ():line 2692, which is
    2691                 value = get_new_data ();
    2692                 value->len  = vallen;
    2693                 value->data = memdup (buf, vallen);
    2694                 value->is_static = 0;
    2695                 buf += vallen;
    2696
    2697                 dict_add (*fill, key, value);

As we can see, we don't check if "value" has got any memory allocated. Looks like mem_get0 ran out of pool space, and returned null, inturn forcing
get_new_data() to return null. Dereferencing "value" without checking caused the crash.

Regards,
Avra

On 11/21/2014 05:46 PM, Rajesh Joseph wrote:
> ----- Original Message -----
>> From: "Atin Mukherjee" <amukherj at redhat.com>
>> To: "Rajesh Joseph" <rjoseph at redhat.com>, "Avra Sengupta" <asengupt at redhat.com>
>> Cc: a-team at redhat.com, gluster-devel at gluster.org
>> Sent: Friday, November 21, 2014 2:07:28 PM
>> Subject: USS test cases failure with core
>>
>> Rajesh/Avra,
>>
>> For one of the regression run [1] few uss testcases failed with core file.
>> Can you please take a look?
>>
>> [1]
>> http://build.gluster.org/job/rackspace-regression-2GB-triggered/2821/consoleFull
>>
> Thanks Atin for letting us know. We will look into this and revert back.
>
> Rajesh