[Bugs] [Bug 1466037] Fuse mount crashed with continuous dd on a file and reading the file in parallel

bugzilla at redhat.com bugzilla at redhat.com
Thu Jun 29 16:23:31 UTC 2017


https://bugzilla.redhat.com/show_bug.cgi?id=1466037



--- Comment #4 from Worker Ant <bugzilla-bot at gluster.org> ---
COMMIT: https://review.gluster.org/17644 committed in master by Pranith Kumar
Karampuri (pkarampu at redhat.com) 
------
commit 97defef2375b911c7b6a3924c242ba8ef4593686
Author: Pranith Kumar K <pkarampu at redhat.com>
Date:   Wed Jun 28 09:10:53 2017 +0530

    features/shard: Remove ctx from LRU in shard_forget

    Problem:
    There is a race when the following two commands are executed on the mount
in
    parallel from two different terminals on a sharded volume,
    which leads to use-after-free.

    Terminal-1:
    while true; do dd if=/dev/zero of=file1 bs=1M count=4; done

    Terminal-2:
    while true; do cat file1 > /dev/null; done

    In the normal case this is the life-cycle of a shard-inode
    1) Shard is added to LRU when it is first looked-up
    2) For every operation on the shard it is moved up in LRU
    3) When "unlink of the shard"/"LRU limit is hit" happens it is removed from
LRU

    But we are seeing a race where the inode stays in Shard LRU even after it
is
    forgotten which leads to Use-after-free and then some memory-corruptions.

    These are the steps:
    1) Shard is added to LRU when it is first looked-up
    2) For every operation on the shard it is moved up in LRU

    Reader-handler                                    Truncate-handler
    1) Reader handler needs shard-x to be read.     1) Truncate has just
deleted shard-x
    2) In shard_common_resolve_shards(), it does
       inode_resolve() and that leads to
       a hit in LRU, so it is going to call
    __shard_update_shards_inode_list() to move the
       inode to top of LRU
                            2) shard-x gets unlinked from the itable
                               and inode_forget(inode, 0) is called
                               to make sure the inode can be purged
                               upon last unref
    3) when __shard_update_shards_inode_list() is
       called it finds that the inode is not in LRU
       so it adds it back to the LRU-list

    Both these operations complete and call inode_unref(shard-x) which leads to
the inode
    getting freed and forgotten, even when it is in Shard LRU list. When more
inodes are
    added to LRU, use-after-free will happen and it leads to undefined
behaviors.

    Fix:
    I see that the inode can be removed from LRU even by the protocol layers
like gfapi/gNFS
    when LRU limit is reached. So it is better to add a check in shard_forget()
to remove itself
    from LRU list if it exists.

    BUG: 1466037
    Change-Id: Ia79c0c5c9d5febc56c41ddb12b5daf03e5281638
    Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>
    Reviewed-on: https://review.gluster.org/17644
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Krutika Dhananjay <kdhananj at redhat.com>

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.


More information about the Bugs mailing list