[Gluster-users] Readdirp (ls -l) Performance Improvement

[Yaniv Kaul]

On Wed, May 27, 2020 at 10:35 AM RAFI KC <rkavunga at redhat.com> wrote:

> Hi Felix,
>
> Thanks for your mail. I will test it more to make sure that it doesn't
> break anything. Also I have added a configuration key for easier switching
> to the older code in case if there is any problem. If you can help me in
> any manner in testing or performance numbers, please let me know.
>

A scratch build allowing the community to test this might be beneficial.
Y.

>
> Regards
>
> Rafi KC
> On 27/05/20 12:37 pm, Felix Kölzow wrote:
>
> Dear Rafi,
>
>
> thanks for your effort. I think this is of great interest of many gluster
> users. Thus, I would really encourage you to
>
> test and to further improve this feature. Maybe it is beneficial to create
> a certain guideline which things should be tested
>
> to make this feature really ready for productive use.
>
>
> Thanks in advance.
>
> Felix
> On 27/05/2020 07:56, RAFI KC wrote:
>
> Hi All,
>
> I have been working on POC to improve readdirp performance improvement. At
> the end of the experiment, The results are showing promising result in
> performance, overall there is a 104% improvement for full filesystem crawl
> compared to the existing solution. Here is the short test numbers. The
> tests were carried out in 16*3 setup with 1.5 Million dentries (Both files
> and dir). The system also contains some empty directories. *In the result
> the proposed solution is 287% faster than the plane volume and 104% faster
> than the parallel-readdir based solution.*
>
>
> Configuration
>
> Plain volume
>
> Parallel-readdir
>
> Proposed Solution
>
> FS Crawl Time in Seconds
>
> 16497.523
>
> 8717.872
>
> 4261.401
>
> In short, the basic idea behind the proposal is the efficient managing of
> readdir buffer in gluster along with prefetching the dentries for
> intelligent switch-over to the next buffer. The detailed problem
> description, deign description and results are available in the doc.
> https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit
>
>
> If anybody can help with the testing on a different kind of workloads, I
> would be very happy to assist. If wanted to test the patch and run a
> performance test on your setup, I could help with back-porting the patch to
> the version of your choice.
>
>
> https://review.gluster.org/24469
>
> https://review.gluster.org/24470
>
>
> Regards
>
> Rafi KC
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200527/5da6989c/attachment.html>