[Bugs] [Bug 1364420] [RFE] History Crawl performance improvement

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1364420



--- Comment #13 from Worker Ant <bugzilla-bot at gluster.org> ---
COMMIT: http://review.gluster.org/15110 committed in master by Aravinda VK
(avishwan at redhat.com) 
------
commit 6c283f107b646405936520e2549510115bf2ef64
Author: Aravinda VK <avishwan at redhat.com>
Date:   Mon Aug 8 17:02:37 2016 +0530

    geo-rep: Post process Data and Meta Changelogs

    With this patch, Data and Meta GFIDs are post processed. If Changelog has
    UNLINK entry then remove from Data and Meta GFIDs list(If stat on GFID is
    ENOENT in Master).

    While processing Changelogs,

    - Collect all the data and meta operations in a temporary database
    - Delete all Data and Meta GFIDs which are already unlinked as per
Changelogs
      (unlink only if stat on GFID is ENOENT)
    - Process all Entry operations as usual
    - Process data and meta operations in batch(Fetch from Db in batch)
    - Data sync is again batched based on number of changelogs(Default 1day
      changelogs). Once the sync is complete, Update last Changelog's time as
last_synced
      time as usual.

    Additionally maintain entry_stime on Brick root, ignore Entry ops if
changelog
    suffix time is less than entry_stime. If data stime is more than
entry_stime,
    this can happen only when passive worker updates stime by itself by getting
    mount point stime. Use entry_stime = data_stime in this case.

    New configurations:

    max-rsync-retries - Default Value is 10
    max-data-changelogs-in-batch - Max number of changelogs to be considered in
a
    batch for syncing. Default value is 5760(4 changelogs per min * 60 min *
    24 hours)
    max-history-changelogs-in-batch - Max number of history changelogs to be
    processed at once. Default value 86400(4 changelogs per min * 60 min * 24
    hours * 15 days)

    BUG: 1364420
    Change-Id: I7b665895bf4806035c2a8573d361257cbadbea17
    Signed-off-by: Aravinda VK <avishwan at redhat.com>
    Reviewed-on: http://review.gluster.org/15110
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    Reviewed-by: Kotresh HR <khiremat at redhat.com>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=OqeoCQMr9j&a=cc_unsubscribe