[Bugs] [Bug 1288921] New: Use after free bug in notify_kernel_loop in fuse-bridge code

bugzilla at redhat.com bugzilla at redhat.com
Mon Dec 7 02:18:54 UTC 2015 

https://bugzilla.redhat.com/show_bug.cgi?id=1288921

            Bug ID: 1288921
           Summary: Use after free bug in notify_kernel_loop in
                    fuse-bridge code
           Product: Red Hat Gluster Storage
           Version: 3.1
         Component: glusterfs-fuse
          Keywords: ZStream
          Assignee: pkarampu at redhat.com
          Reporter: pkarampu at redhat.com
        QA Contact: storage-qa-internal at redhat.com
                CC: bugs at gluster.org, chrisw at redhat.com, csaba at redhat.com,
                    gluster-bugs at redhat.com, nlevinki at redhat.com
        Depends On: 1288857
          Keywords: ZStream



+++ This bug was initially created as a clone of Bug #1288857 +++

Description of problem:
    fouh->len is accessed after 'node' is freed. Also rv is int where as
    fouh->len is uint32 comparison needs to be changed to ssize_t variables.

Asan report:
==10762== ERROR: AddressSanitizer: heap-use-after-free on address
0x602c00048700 at pc 0x7f667e468a00 bp 0x7f6675c42e20 sp 0x7f6675c42e10
READ of size 4 at 0x602c00048700 thread T9
    #0 0x7f667e4689ff in notify_kernel_loop
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3875
    #1 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
    #2 0x3cf4207ee4 in start_thread (/lib64/libpthread.so.0+0x3cf4207ee4)
    #3 0x3cf3ef4d1c in __clone (/lib64/libc.so.6+0x3cf3ef4d1c)
0x602c00048700 is located 64 bytes inside of 376-byte region
[0x602c000486c0,0x602c00048838)
freed by thread T9 here:
    #0 0x7f66860e00f9 (/lib64/libasan.so.0+0x160f9)
    #1 0x7f6685d5e6a4 in __gf_free
/home/pk1/workspace/gerrit-repo/libglusterfs/src/mem-pool.c:336
    #2 0x7f667e4689c4 in notify_kernel_loop
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3873
    #3 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
previously allocated by thread T7 here:
    #0 0x7f66860e0315 (/lib64/libasan.so.0+0x16315)
    #1 0x7f6685d5d3be in __gf_calloc
/home/pk1/workspace/gerrit-repo/libglusterfs/src/mem-pool.c:117
    #2 0x7f667e4308b7 in fuse_invalidate_inode
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:295
    #3 0x7f667e42f61c in fuse_invalidate
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:55
    #4 0x7f6685d22071 in inode_invalidate
/home/pk1/workspace/gerrit-repo/libglusterfs/src/inode.c:1158
    #5 0x7f66790789ed in mdc_inode_iatt_set_validate
/home/pk1/workspace/gerrit-repo/xlators/performance/md-cache/src/md-cache.c:427
    #6 0x7f667907e5da in mdc_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/performance/md-cache/src/md-cache.c:1040
    #7 0x7f6685e3b57c in default_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:1333
    #8 0x7f6685e3b57c in default_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:1333
    #9 0x7f66796d52c6 in ioc_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/performance/io-cache/src/io-cache.c:1327
    #10 0x7f6679b0d33c in ra_truncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/performance/read-ahead/src/read-ahead.c:704
    #11 0x7f6679d38e90 in wb_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/performance/write-behind/src/write-behind.c:1693
    #12 0x7f667a02a74e in dht_truncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/cluster/dht/src/dht-inode-write.c:283
    #13 0x7f667a2ee5fd in afr_ftruncate_unwind
/home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:646
    #14 0x7f667a2e8200 in __afr_inode_write_cbk
/home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:171
    #15 0x7f667a2ee7a0 in afr_ftruncate_wind_cbk
/home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:665
    #16 0x7f667a610c79 in client3_3_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-rpc-fops.c:1512
    #17 0x7f6685a82e45 in rpc_clnt_handle_reply
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759
    #18 0x7f6685a83674 in rpc_clnt_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900
    #19 0x7f6685a7a83a in rpc_transport_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541
    #20 0x7f667b5cda53 in socket_event_poll_in
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231
    #21 0x7f667b5ce720 in socket_event_handler
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344
    #22 0x7f6685ddaf49 in event_dispatch_epoll_handler
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571
    #23 0x7f6685ddb823 in event_dispatch_epoll_worker
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674
    #24 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
Thread T9 created by T8 here:
    #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a)
    #1 0x7f6685d18bf9 in gf_thread_create
/home/pk1/workspace/gerrit-repo/libglusterfs/src/common-utils.c:3468
    #2 0x7f667e4691ee in fuse_init
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3946
    #3 0x7f667e46fc64 in fuse_thread_proc
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:4935
    #4 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
Thread T8 created by T5 here:
    #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a)
    #1 0x7f6685d18bf9 in gf_thread_create
/home/pk1/workspace/gerrit-repo/libglusterfs/src/common-utils.c:3468
    #2 0x7f667e471205 in notify
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:5170
    #3 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #4 0x7f6685e58f97 in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2879
    #5 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #6 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
    #7 0x7f6678e5e4bb in notify
/home/pk1/workspace/gerrit-repo/xlators/debug/io-stats/src/io-stats.c:3838
    #8 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #9 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
    #10 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #11 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
    #12 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #13 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
    #14 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #15 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
    #16 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #17 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
    #18 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #19 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
    #20 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #21 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
    #22 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #23 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
    #24 0x7f667a024ddc in dht_notify
/home/pk1/workspace/gerrit-repo/xlators/cluster/dht/src/dht-common.c:7888
    #25 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #26 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
    #27 0x7f667a38f3ff in afr_notify
/home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-common.c:4021
    #28 0x7f667a3968be in notify
/home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr.c:34
    #29 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
    #30 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
    #31 0x7f667a5dc91a in client_notify_dispatch
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:83
    #32 0x7f667a5dc761 in client_notify_dispatch_uniq
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:61
    #33 0x7f667a64f7d2 in client_notify_parents_child_up
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:133
    #34 0x7f667a65551a in client_post_handshake
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:1053
    #35 0x7f667a65637b in client_setvolume_cbk
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:1210
    #36 0x7f6685a82e45 in rpc_clnt_handle_reply
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759
    #37 0x7f6685a83674 in rpc_clnt_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900
    #38 0x7f6685a7a83a in rpc_transport_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541
    #39 0x7f667b5cda53 in socket_event_poll_in
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231
    #40 0x7f667b5ce720 in socket_event_handler
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344
    #41 0x7f6685ddaf49 in event_dispatch_epoll_handler
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571
    #42 0x7f6685ddb823 in event_dispatch_epoll_worker
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674
    #43 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
Thread T5 created by T0 here:
    #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a)
    #1 0x7f6685ddba89 in event_dispatch_epoll
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:726
    #2 0x7f6685d5b92f in event_dispatch
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event.c:124
    #3 0x40eeb6 in main
/home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd.c:2345
    #4 0x3cf3e21d64 in __libc_start_main (/lib64/libc.so.6+0x3cf3e21d64)
Thread T7 created by T5 here:
    #0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a)
    #1 0x7f6685ddbfac in event_reconfigure_threads_epoll
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:834
    #2 0x7f6685d5ba8b in event_reconfigure_threads
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event.c:140
    #3 0x7f667a5f5f6c in client_check_event_threads
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:2332
    #4 0x7f667a5f69ec in init
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:2448
    #5 0x7f6685cf665d in __xlator_init
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:399
    #6 0x7f6685cf68b7 in xlator_init
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:424
    #7 0x7f6685d83a14 in glusterfs_graph_init
/home/pk1/workspace/gerrit-repo/libglusterfs/src/graph.c:320
    #8 0x7f6685d84dec in glusterfs_graph_activate
/home/pk1/workspace/gerrit-repo/libglusterfs/src/graph.c:667
    #9 0x40e4f4 in glusterfs_process_volfp
/home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd.c:2186
    #10 0x417168 in mgmt_getspec_cbk
/home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd-mgmt.c:1640
    #11 0x7f6685a82e45 in rpc_clnt_handle_reply
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759
    #12 0x7f6685a83674 in rpc_clnt_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900
    #13 0x7f6685a7a83a in rpc_transport_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541
    #14 0x7f667b5cda53 in socket_event_poll_in
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231
    #15 0x7f667b5ce720 in socket_event_handler
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344
    #16 0x7f6685ddaf49 in event_dispatch_epoll_handler
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571
    #17 0x7f6685ddb823 in event_dispatch_epoll_worker
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674
    #18 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
SUMMARY: AddressSanitizer: heap-use-after-free
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3875
notify_kernel_loop
Shadow bytes around the buggy address:
  0x0c0600001090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
  0x0c06000010a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c06000010b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c06000010c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c06000010d0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x0c06000010e0:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c06000010f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c0600001100: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
  0x0c0600001110: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c0600001120: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c0600001130: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==10762== ABORTING
fsync: Software caused connection abort

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. run iozone -a on a mount with address sanitizer enabled build and it
crashes.
2.
3.

Actual results:


Expected results:


Additional info:

--- Additional comment from Vijay Bellur on 2015-12-06 12:21:03 EST ---

REVIEW: http://review.gluster.org/12886 (mount/fuse: Fix use-after-free crash)
posted (#1) for review on master by Pranith Kumar Karampuri
(pkarampu at redhat.com)


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1288857
[Bug 1288857] Use after free bug in notify_kernel_loop in fuse-bridge code
-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=PRWoHTF3wE&a=cc_unsubscribe

More information about the Bugs mailing list