[Gluster-users] Readdirp (ls -l) Performance Improvement

[RAFI KC]

Result for a single ls on a directory with 10k directories inside (16*3 
volume)

*
*

Configuration

	

Plain volume

	

Parallel-readdir

	

Proposed Solution

Single Dir ls (Seconds)

	

-----

	

135

	

32.744


It is showing 321% improvements.

Regards
Rafi KC
On 27/05/20 11:26 am, RAFI KC wrote:
>
> Hi All,
>
> I have been working on POC to improve readdirp performance 
> improvement. At the end of the experiment, The results are showing 
> promising result in performance, overall there is a 104% improvement 
> for full filesystem crawl compared to the existing solution. Here is 
> the short test numbers. The tests were carried out in 16*3 setup with 
> 1.5 Million dentries (Both files and dir). The system also contains 
> some empty directories. *In the result the proposed solution is 287% 
> faster than the plane volume and 104% faster than the parallel-readdir 
> based solution.*
>
> *
> *
>
> Configuration
>
> 	
>
> Plain volume
>
> 	
>
> Parallel-readdir
>
> 	
>
> Proposed Solution
>
> FS Crawl Time in Seconds
>
> 	
>
> 16497.523
>
> 	
>
> 8717.872
>
> 	
>
> 4261.401
>
> In short, the basic idea behind the proposal is the efficient managing 
> of readdir buffer in gluster along with prefetching the dentries for 
> intelligent switch-over to the next buffer. The detailed problem 
> description, deign description and results are available in the 
> doc.https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit 
>
>
>
> If anybody can help with the testing on a different kind of workloads, 
> I would be very happy to assist. If wanted to test the patch and run a 
> performance test on your setup, I could help with back-porting the 
> patch to the version of your choice.
>
>
> https://review.gluster.org/24469
>
> https://review.gluster.org/24470
>
>
> Regards
>
> Rafi KC
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200527/6c27681a/attachment.html>