[Gluster-devel] Multithreaded Iterative Dir Tree Scan

[sankarshan]

On Fri, 24 Apr 2020 at 08:05, Amar Tumballi <amar at kadalu.io> wrote:

> This looks like a good effort to pick up Barak. A needed one indeed.
>
>
Should this be tracked with a release label and planned? The content of the
document should probably transfer itself to the issue tracking the PR(s)



> On Mon, Mar 23, 2020 at 3:18 PM Barak Sason Rofman <bsasonro at redhat.com>
> wrote:
>
>> Hello everyone!
>> Following a discussion I had with @Susant Palai some time ago, we have
>> decided to look into an option to improve the rebalance process in the DHT
>> layer by modifying the underlying mechanism. Currently, dir-tree crawling
>> is done recursively, by a single thread, which is likely slow and also
>> poses the risk of stack overflow. An iterative multithreaded solution might
>> improve performance and also stability (by eliminating the risk of stack
>> overflow). I have prepared a POC doc on the matter, including a sample
>> implementation of the iterative multithreaded solution. The doc can be
>> found at:
>>
>> https://docs.google.com/document/d/1JCl0T9zeagOcFFpgVQF8zNyhlR54VqkNAZ7TJb42egE/edit
>>
>> <https://docs.google.com/document/d/1L0uHgFbrNWWxCQB6s4YcoymKrO7q0yVAbEIWWIiu_as/edit?usp=sharing>Apart
>> from the rebalance process, maybe this approach can be useful for other
>> use-cases where dir-tree crawl is being performed? Any comments on the
>> concept, the design of the solution and the implementation are welcome.
>>
>> --
>> *Barak Sason Rofman*
>>
>> Gluster Storage Development
>>
>> Red Hat Israel <https://www.redhat.com/>
>>
>> 34 Jerusalem rd. Ra'anana, 43501
>>
>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>> M: *+972-52-4326355*
>>
>

-- 
sankarshan mukhopadhyay
<https://about.me/sankarshan.mukhopadhyay>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20200424/39fe6148/attachment.html>