[Gluster-devel] memory corruption in release-3.3

[Emmanuel Dreyfus]

Hi

I still get crashes caused by memory corruption with latest release-3.3. 
My test case is a rm -Rf on a large tree. It seems I crash in two places:

First crash flavor (trav is sometimes unmapped memory, sometimes NULL)

#0  0xbbbb60ad in __iobuf_select_arena (iobuf_pool=0xbb70d400, page_size=128)
    at iobuf.c:453
453                     if (trav->passive_cnt) {
(gdb) print trav
$1 = (struct iobuf_arena *) 0x414d202c
(gdb) bt
#0  0xbbbb60ad in __iobuf_select_arena (iobuf_pool=0xbb70d400, page_size=128)
    at iobuf.c:453
#1  0xbbbb655a in iobuf_get2 (iobuf_pool=0xbb70d400, page_size=24)
    at iobuf.c:604
#2  0xbaa549c7 in client_submit_request ()
   from /usr/local/lib/glusterfs/3.3git/xlator/protocol/client.so
#3  0xbaa732c5 in client3_1_open ()
   from /usr/local/lib/glusterfs/3.3git/xlator/protocol/client.so
#4  0xbaa574e6 in client_open ()
   from /usr/local/lib/glusterfs/3.3git/xlator/protocol/client.so
#5  0xb9abac10 in afr_sh_data_open ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#6  0xb9abacb9 in afr_self_heal_data ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#7  0xb9ac2751 in afr_sh_metadata_done ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#8  0xb9ac457a in afr_self_heal_metadata ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#9  0xb9abd93f in afr_sh_missing_entries_done ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#10 0xb9ac169b in afr_self_heal ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#11 0xb9ae2e5b in afr_launch_self_heal ()
#12 0xb9ae3de9 in afr_lookup_perform_self_heal ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#13 0xb9ae4804 in afr_lookup_done ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#14 0xb9ae4fab in afr_lookup_cbk ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#15 0xbaa6dc10 in client3_1_lookup_cbk ()
   from /usr/local/lib/glusterfs/3.3git/xlator/protocol/client.so
#16 0xbbb69716 in rpc_clnt_handle_reply () from /usr/local/lib/libgfrpc.so.0
#17 0xbbb699b3 in rpc_clnt_notify () from /usr/local/lib/libgfrpc.so.0
#18 0xbbb65989 in rpc_transport_notify () from /usr/local/lib/libgfrpc.so.0
#19 0xbaa9327e in socket_event_poll_in ()
   from /usr/local/lib/glusterfs/3.3git/rpc-transport/socket.so
#20 0xbaa937f5 in socket_event_handler ()
   from /usr/local/lib/glusterfs/3.3git/rpc-transport/socket.so
#21 0xbbbb270f in event_dispatch_poll_handler (event_pool=0xbb73b080, 
    ufds=0xbb77e6a0, i=3) at event.c:357
#22 0xbbbb297b in event_dispatch_poll (event_pool=0xbb73b080) at event.c:437
#23 0xbbbb2ca7 in event_dispatch (event_pool=0xbb73b080) at event.c:947
#24 0x08050078 in main ()


Second crash flavor (it looks more like a double free)

Program terminated with signal 11, Segmentation fault.
#0  0xbb92661e in ?? () from /lib/libc.so.12
(gdb) bt
#0  0xbb92661e in ?? () from /lib/libc.so.12
#1  0xbb92891b in free () from /lib/libc.so.12
#2  0xbbbb376f in __gf_free (free_ptr=0xbb70d160) at mem-pool.c:258
#3  0xbbb7e17d in data_destroy (data=0xba301d4c) at dict.c:135
#4  0xbbb7ee18 in data_unref (this=0xba301d4c) at dict.c:470
#5  0xbbb7eb6b in dict_destroy (this=0xba4022d0) at dict.c:395
#6  0xbbb7ecab in dict_unref (this=0xba4022d0) at dict.c:432
#7  0xbaa164ba in __qr_inode_free ()
   from /usr/local/lib/glusterfs/3.3git/xlator/performance/quick-read.so
#8  0xbaa27164 in qr_forget ()
   from /usr/local/lib/glusterfs/3.3git/xlator/performance/quick-read.so
#9  0xbbb9b221 in __inode_destroy (inode=0xb8b017e4) at inode.c:320
#10 0xbbb9d0a5 in inode_table_prune (table=0xba3cc160) at inode.c:1235
#11 0xbbb9b64e in inode_unref (inode=0xb8b017e4) at inode.c:445
#12 0xbbb85249 in loc_wipe (loc=0xb9402dd0) at xlator.c:530
#13 0xb9ae126e in afr_local_cleanup ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#14 0xb9a9c66b in afr_unlink_done ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#15 0xb9ad2d5b in afr_unlock_common_cbk ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
#16 0xb9ad38a2 in afr_unlock_entrylk_cbk ()
   from /usr/local/lib/glusterfs/3.3git/xlator/cluster/replicate.so
---Type <return> to continue, or q <return> to quit---
#17 0xbaa68370 in client3_1_entrylk_cbk ()
   from /usr/local/lib/glusterfs/3.3git/xlator/protocol/client.so
#18 0xbbb69716 in rpc_clnt_handle_reply () from /usr/local/lib/libgfrpc.so.0
#19 0xbbb699b3 in rpc_clnt_notify () from /usr/local/lib/libgfrpc.so.0
#20 0xbbb65989 in rpc_transport_notify () from /usr/local/lib/libgfrpc.so.0
#21 0xbaa9327e in socket_event_poll_in ()
   from /usr/local/lib/glusterfs/3.3git/rpc-transport/socket.so
#22 0xbaa937f5 in socket_event_handler ()
   from /usr/local/lib/glusterfs/3.3git/rpc-transport/socket.so
#23 0xbbbb270f in event_dispatch_poll_handler (event_pool=0xbb73b080, 
    ufds=0xbb77e6a0, i=3) at event.c:357
#24 0xbbbb297b in event_dispatch_poll (event_pool=0xbb73b080) at event.c:437
#25 0xbbbb2ca7 in event_dispatch (event_pool=0xbb73b080) at event.c:947
#26 0x08050078 in main ()
(gdb) frame 2
#2  0xbbbb376f in __gf_free (free_ptr=0xbb70d160) at mem-pool.c:258
258                     FREE (free_ptr);
(gdb) x/1w free_ptr
0xbb70d160:     538978863


-- 
Emmanuel Dreyfus
manu at netbsd.org