[Gluster-devel] Multithreaded Iterative Dir Tree Scan

[Barak Sason Rofman]

Hello everyone!
Following a discussion I had with @Susant Palai some time ago, we have
decided to look into an option to improve the rebalance process in the DHT
layer by modifying the underlying mechanism. Currently, dir-tree crawling
is done recursively, by a single thread, which is likely slow and also
poses the risk of stack overflow. An iterative multithreaded solution might
improve performance and also stability (by eliminating the risk of stack
overflow). I have prepared a POC doc on the matter, including a sample
implementation of the iterative multithreaded solution. The doc can be
found at:
https://docs.google.com/document/d/1JCl0T9zeagOcFFpgVQF8zNyhlR54VqkNAZ7TJb42egE/edit
<https://docs.google.com/document/d/1L0uHgFbrNWWxCQB6s4YcoymKrO7q0yVAbEIWWIiu_as/edit?usp=sharing>Apart
from the rebalance process, maybe this approach can be useful for other
use-cases where dir-tree crawl is being performed? Any comments on the
concept, the design of the solution and the implementation are welcome.

-- 
*Barak Sason Rofman*

Gluster Storage Development

Red Hat Israel <https://www.redhat.com/>

34 Jerusalem rd. Ra'anana, 43501

bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
M: *+972-52-4326355*
<https://red.ht/sig>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20200323/3f48a617/attachment.html>