[Bugs] [Bug 1327036] New: Use after free bug in notify_kernel_loop in fuse-bridge code
bugzilla at redhat.com
bugzilla at redhat.com
Thu Apr 14 07:07:50 UTC 2016
https://bugzilla.redhat.com/show_bug.cgi?id=1327036
Bug ID: 1327036
Summary: Use after free bug in notify_kernel_loop in
fuse-bridge code
Product: Red Hat Gluster Storage
Version: 3.1
Component: glusterfs-fuse
Assignee: pkarampu at redhat.com
Reporter: pkarampu at redhat.com
QA Contact: storage-qa-internal at redhat.com
CC: bugs at gluster.org, chrisw at redhat.com, csaba at redhat.com,
nlevinki at redhat.com, rgowdapp at redhat.com
Depends On: 1288857
Blocks: 1288922, 1288921
+++ This bug was initially created as a clone of Bug #1288857 +++
Description of problem:
fouh->len is accessed after 'node' is freed. Also rv is int where as
fouh->len is uint32 comparison needs to be changed to ssize_t variables.
Asan report:
==10762== ERROR: AddressSanitizer: heap-use-after-free on address
0x602c00048700 at pc 0x7f667e468a00 bp 0x7f6675c42e20 sp 0x7f6675c42e10
READ of size 4 at 0x602c00048700 thread T9
#0 0x7f667e4689ff in notify_kernel_loop
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3875
#1 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
#2 0x3cf4207ee4 in start_thread (/lib64/libpthread.so.0+0x3cf4207ee4)
#3 0x3cf3ef4d1c in __clone (/lib64/libc.so.6+0x3cf3ef4d1c)
0x602c00048700 is located 64 bytes inside of 376-byte region
[0x602c000486c0,0x602c00048838)
freed by thread T9 here:
#0 0x7f66860e00f9 (/lib64/libasan.so.0+0x160f9)
#1 0x7f6685d5e6a4 in __gf_free
/home/pk1/workspace/gerrit-repo/libglusterfs/src/mem-pool.c:336
#2 0x7f667e4689c4 in notify_kernel_loop
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3873
#3 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
previously allocated by thread T7 here:
#0 0x7f66860e0315 (/lib64/libasan.so.0+0x16315)
#1 0x7f6685d5d3be in __gf_calloc
/home/pk1/workspace/gerrit-repo/libglusterfs/src/mem-pool.c:117
#2 0x7f667e4308b7 in fuse_invalidate_inode
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:295
#3 0x7f667e42f61c in fuse_invalidate
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:55
#4 0x7f6685d22071 in inode_invalidate
/home/pk1/workspace/gerrit-repo/libglusterfs/src/inode.c:1158
#5 0x7f66790789ed in mdc_inode_iatt_set_validate
/home/pk1/workspace/gerrit-repo/xlators/performance/md-cache/src/md-cache.c:427
#6 0x7f667907e5da in mdc_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/performance/md-cache/src/md-cache.c:1040
#7 0x7f6685e3b57c in default_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:1333
#8 0x7f6685e3b57c in default_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:1333
#9 0x7f66796d52c6 in ioc_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/performance/io-cache/src/io-cache.c:1327
#10 0x7f6679b0d33c in ra_truncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/performance/read-ahead/src/read-ahead.c:704
#11 0x7f6679d38e90 in wb_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/performance/write-behind/src/write-behind.c:1693
#12 0x7f667a02a74e in dht_truncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/cluster/dht/src/dht-inode-write.c:283
#13 0x7f667a2ee5fd in afr_ftruncate_unwind
/home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:646
#14 0x7f667a2e8200 in __afr_inode_write_cbk
/home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:171
#15 0x7f667a2ee7a0 in afr_ftruncate_wind_cbk
/home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-inode-write.c:665
#16 0x7f667a610c79 in client3_3_ftruncate_cbk
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-rpc-fops.c:1512
#17 0x7f6685a82e45 in rpc_clnt_handle_reply
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759
#18 0x7f6685a83674 in rpc_clnt_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900
#19 0x7f6685a7a83a in rpc_transport_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541
#20 0x7f667b5cda53 in socket_event_poll_in
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231
#21 0x7f667b5ce720 in socket_event_handler
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344
#22 0x7f6685ddaf49 in event_dispatch_epoll_handler
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571
#23 0x7f6685ddb823 in event_dispatch_epoll_worker
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674
#24 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
Thread T9 created by T8 here:
#0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a)
#1 0x7f6685d18bf9 in gf_thread_create
/home/pk1/workspace/gerrit-repo/libglusterfs/src/common-utils.c:3468
#2 0x7f667e4691ee in fuse_init
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3946
#3 0x7f667e46fc64 in fuse_thread_proc
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:4935
#4 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
Thread T8 created by T5 here:
#0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a)
#1 0x7f6685d18bf9 in gf_thread_create
/home/pk1/workspace/gerrit-repo/libglusterfs/src/common-utils.c:3468
#2 0x7f667e471205 in notify
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:5170
#3 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#4 0x7f6685e58f97 in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2879
#5 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#6 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
#7 0x7f6678e5e4bb in notify
/home/pk1/workspace/gerrit-repo/xlators/debug/io-stats/src/io-stats.c:3838
#8 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#9 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
#10 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#11 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
#12 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#13 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
#14 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#15 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
#16 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#17 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
#18 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#19 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
#20 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#21 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
#22 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#23 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
#24 0x7f667a024ddc in dht_notify
/home/pk1/workspace/gerrit-repo/xlators/cluster/dht/src/dht-common.c:7888
#25 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#26 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
#27 0x7f667a38f3ff in afr_notify
/home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr-common.c:4021
#28 0x7f667a3968be in notify
/home/pk1/workspace/gerrit-repo/xlators/cluster/afr/src/afr.c:34
#29 0x7f6685cf6faa in xlator_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:492
#30 0x7f6685e5903b in default_notify
/home/pk1/workspace/gerrit-repo/libglusterfs/src/defaults.c:2885
#31 0x7f667a5dc91a in client_notify_dispatch
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:83
#32 0x7f667a5dc761 in client_notify_dispatch_uniq
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:61
#33 0x7f667a64f7d2 in client_notify_parents_child_up
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:133
#34 0x7f667a65551a in client_post_handshake
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:1053
#35 0x7f667a65637b in client_setvolume_cbk
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client-handshake.c:1210
#36 0x7f6685a82e45 in rpc_clnt_handle_reply
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759
#37 0x7f6685a83674 in rpc_clnt_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900
#38 0x7f6685a7a83a in rpc_transport_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541
#39 0x7f667b5cda53 in socket_event_poll_in
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231
#40 0x7f667b5ce720 in socket_event_handler
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344
#41 0x7f6685ddaf49 in event_dispatch_epoll_handler
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571
#42 0x7f6685ddb823 in event_dispatch_epoll_worker
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674
#43 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
Thread T5 created by T0 here:
#0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a)
#1 0x7f6685ddba89 in event_dispatch_epoll
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:726
#2 0x7f6685d5b92f in event_dispatch
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event.c:124
#3 0x40eeb6 in main
/home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd.c:2345
#4 0x3cf3e21d64 in __libc_start_main (/lib64/libc.so.6+0x3cf3e21d64)
Thread T7 created by T5 here:
#0 0x7f66860d4d2a (/lib64/libasan.so.0+0xad2a)
#1 0x7f6685ddbfac in event_reconfigure_threads_epoll
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:834
#2 0x7f6685d5ba8b in event_reconfigure_threads
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event.c:140
#3 0x7f667a5f5f6c in client_check_event_threads
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:2332
#4 0x7f667a5f69ec in init
/home/pk1/workspace/gerrit-repo/xlators/protocol/client/src/client.c:2448
#5 0x7f6685cf665d in __xlator_init
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:399
#6 0x7f6685cf68b7 in xlator_init
/home/pk1/workspace/gerrit-repo/libglusterfs/src/xlator.c:424
#7 0x7f6685d83a14 in glusterfs_graph_init
/home/pk1/workspace/gerrit-repo/libglusterfs/src/graph.c:320
#8 0x7f6685d84dec in glusterfs_graph_activate
/home/pk1/workspace/gerrit-repo/libglusterfs/src/graph.c:667
#9 0x40e4f4 in glusterfs_process_volfp
/home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd.c:2186
#10 0x417168 in mgmt_getspec_cbk
/home/pk1/workspace/gerrit-repo/glusterfsd/src/glusterfsd-mgmt.c:1640
#11 0x7f6685a82e45 in rpc_clnt_handle_reply
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:759
#12 0x7f6685a83674 in rpc_clnt_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-clnt.c:900
#13 0x7f6685a7a83a in rpc_transport_notify
/home/pk1/workspace/gerrit-repo/rpc/rpc-lib/src/rpc-transport.c:541
#14 0x7f667b5cda53 in socket_event_poll_in
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2231
#15 0x7f667b5ce720 in socket_event_handler
/home/pk1/workspace/gerrit-repo/rpc/rpc-transport/socket/src/socket.c:2344
#16 0x7f6685ddaf49 in event_dispatch_epoll_handler
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:571
#17 0x7f6685ddb823 in event_dispatch_epoll_worker
/home/pk1/workspace/gerrit-repo/libglusterfs/src/event-epoll.c:674
#18 0x7f66860e3bb7 (/lib64/libasan.so.0+0x19bb7)
SUMMARY: AddressSanitizer: heap-use-after-free
/home/pk1/workspace/gerrit-repo/xlators/mount/fuse/src/fuse-bridge.c:3875
notify_kernel_loop
Shadow bytes around the buggy address:
0x0c0600001090: fd fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa
0x0c06000010a0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c06000010b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c06000010c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c06000010d0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x0c06000010e0:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c06000010f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c0600001100: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
0x0c0600001110: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c0600001120: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c0600001130: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap righ redzone: fb
Freed Heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
ASan internal: fe
==10762== ABORTING
fsync: Software caused connection abort
Version-Release number of selected component (if applicable):
How reproducible:
Steps to Reproduce:
1. run iozone -a on a mount with address sanitizer enabled build and it
crashes.
2.
3.
Actual results:
Expected results:
Additional info:
--- Additional comment from Vijay Bellur on 2015-12-06 12:21:03 EST ---
REVIEW: http://review.gluster.org/12886 (mount/fuse: Fix use-after-free crash)
posted (#1) for review on master by Pranith Kumar Karampuri
(pkarampu at redhat.com)
--- Additional comment from Vijay Bellur on 2015-12-06 23:57:41 EST ---
COMMIT: http://review.gluster.org/12886 committed in master by Raghavendra G
(rgowdapp at redhat.com)
------
commit 05b510bb893761864d3830eb781210445056a6f9
Author: Pranith Kumar K <pkarampu at redhat.com>
Date: Sun Dec 6 22:05:54 2015 +0530
mount/fuse: Fix use-after-free crash
fouh->len is accessed after 'node' is freed. Also 'rv' is int where as
fouh->len is uint32, changed comparison to ssize_t variables.
BUG: 1288857
Change-Id: Ied43d29e1e52719f9b52fe839cee31ce65711eea
Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>
Reviewed-on: http://review.gluster.org/12886
Tested-by: Gluster Build System <jenkins at build.gluster.com>
Reviewed-by: Raghavendra G <rgowdapp at redhat.com>
--- Additional comment from Vijay Bellur on 2016-01-21 17:02:20 EST ---
REVIEW: http://review.gluster.org/13274 (fuse: use-after-free fix in
fuse-bridge, revisited) posted (#1) for review on master by Kaleb KEITHLEY
(kkeithle at redhat.com)
--- Additional comment from Vijay Bellur on 2016-02-02 02:23:29 EST ---
REVIEW: http://review.gluster.org/13274 (fuse: use-after-free fix in
fuse-bridge, revisited) posted (#2) for review on master by Kaleb KEITHLEY
(kkeithle at redhat.com)
--- Additional comment from Vijay Bellur on 2016-02-02 05:10:14 EST ---
REVIEW: http://review.gluster.org/13274 (fuse: use-after-free fix in
fuse-bridge, revisited) posted (#3) for review on master by Kaleb KEITHLEY
(kkeithle at redhat.com)
--- Additional comment from Vijay Bellur on 2016-02-03 00:13:14 EST ---
COMMIT: http://review.gluster.org/13274 committed in master by Raghavendra G
(rgowdapp at redhat.com)
------
commit 29bd2316b6d4f522e1bd00e3c9a1c97dcc7d80ea
Author: Kaleb S KEITHLEY <kkeithle at redhat.com>
Date: Thu Jan 21 15:03:38 2016 -0500
fuse: use-after-free fix in fuse-bridge, revisited
Prompted by the email exchange in gluster-devel between Oleksandr
Natalenko, xavi, and soumyak, I looked at this because the fuse client
on the longevity cluster has also been suffering from a serious memory
leak for some time. (longevity cluster is currently running 3.7.6)
The longevity cluster manifests the same kernel notifier loop terminated
log message the Oleksandr sees, and some sample runs suggest that the
length passed to the (sys_)write call is unexpectedly and abnormally large.
Basically this fix
a) uses correct types for len and rv,
b) copies the len from potentially incorrectly aligned memory (in a
way that should minimize potential performance issues related to
accessing unaligned memory.)
c) changes log level of the kernel notifier loop terminated message
d) fixes a potential mutex lock/unlock issue
Change-Id: Icedb3525706f59803878bb37ef6b4ffe4a986880
BUG: 1288857
Signed-off-by: Kaleb S KEITHLEY <kkeithle at redhat.com>
Reviewed-on: http://review.gluster.org/13274
Smoke: Gluster Build System <jenkins at build.gluster.com>
Reviewed-by: Xavier Hernandez <xhernandez at datalab.es>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
Reviewed-by: Raghavendra Bhat <raghavendra at redhat.com>
Reviewed-by: Raghavendra G <rgowdapp at redhat.com>
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1288857
[Bug 1288857] Use after free bug in notify_kernel_loop in fuse-bridge code
https://bugzilla.redhat.com/show_bug.cgi?id=1288921
[Bug 1288921] Use after free bug in notify_kernel_loop in fuse-bridge code
https://bugzilla.redhat.com/show_bug.cgi?id=1288922
[Bug 1288922] Use after free bug in notify_kernel_loop in fuse-bridge code
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=ikgAvZbqtf&a=cc_unsubscribe
More information about the Bugs
mailing list