[Bugs] [Bug 1325857] Multi-threaded SHD support
bugzilla at redhat.com
bugzilla at redhat.com
Sun Apr 17 01:56:19 UTC 2016
--- Comment #3 from Vijay Bellur <vbellur at redhat.com> ---
COMMIT: http://review.gluster.org/13967 committed in release-3.7 by Pranith
Kumar Karampuri (pkarampu at redhat.com)
Author: Pranith Kumar K <pkarampu at redhat.com>
Date: Thu Mar 17 09:32:02 2016 +0530
syncop: Add parallel dir scan functionality
Most of this functionality's ideas are contributed
by Richard Wareing, in his patch:
VERY BIG thanks to him :-).
After starting porting/testing the patch above, I found a few things we can
improve in this patch based on the results we got in testing.
1) We are reading all the indices before we launch self-heals. In some
cases I worked on there were almost 5million files/directories that needed
heal. With such a big number self-heal daemon will be OOM killed if we go
this route. So I modified this to launch heals based on a queue length
2) We found that for directory hierarchies, multi-threaded self-heal
patch was not giving better results compared to single-threaded
self-heal because of the order problems. We improved index xlator to
give gfid type to make sure that all directories in the indices are
healed before the files that follow in that iteration of readdir
output(http://review.gluster.org/13553). In our testing this lead to
zero errors of self-heals as we were only doing self-heals in parallel
for files and not directories. I think we can further improve self-heal
speed for directories by doing name heals in parallel based on similar
techniques Richard's patch showed. I think the best thing there would be to
introduce synccond_t infra (pthread_cond_t kind of infra for syncops)
which I am planning to implement for future releases.
3) Based on 1), 2) and the fact that afr already does retries of the
indices in a loop I removed retries again in the threads.
4) After the refactor, the changes required to bring in multi-threaded
self-heal for ec would just be ~10 lines, most of it will be about
Our tests found that we are able to easily saturate network :-).
High level description of the final feature:
Traditionally self-heal daemon reads the indices (gfids) that need to be
from the brick and initiates heal one gfid at a time. Goal of this feature
to add parallelization to the way we do self-heals in a way we do not
in any case but increase parallelization wherever we can. As part of this
knobs are introduced to improve parallelization:
1) We can launch 'max-jobs' number of heals in parallel.
2) We can keep reading indices as long as the wait-q for heals doesn't go
'max-qlen' passed as arguments to multi-threaded dir_scan.
As a first cut, we always do healing of directories in serial order one at
but for files we launch heals in parallel. In future we can do name-heals
in parallel, but this is not implemented as of now. Reason for this is
already in '2)' above.
AFR/EC can introduce options like max-shd-threads/wait-qlength which can be
by users to increase the rate of heals when they want. Please note that the
options will take effect only for the next crawl.
>Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>
>NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
>CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
>Reviewed-by: Jeff Darcy <jdarcy at redhat.com>
>Smoke: Gluster Build System <jenkins at build.gluster.com>
Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>
Smoke: Gluster Build System <jenkins at build.gluster.com>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=2SBO9hcya7&a=cc_unsubscribe
More information about the Bugs