[Gluster-users] Readdirp (ls -l) Performance Improvement

[RAFI KC]

Hi Felix,

If you already have a test cluster, please give me the gluster version. 
I will try to build rpm then you can just do an upgrade. If you don't 
have a test cluster, and you are creating a new one, then I will build a 
new rpm against the latest head. This is more easier to me as I don't 
have to backport to a previous version.

May I ask you the access protocol that you use (FUSE, SMB , NFS, gfapi). 
I did the testing on a fuse protocol.


Rafi KC

On 27/05/20 4:14 pm, Felix Kölzow wrote:
>
> Dear Rafi KC,
>
>
> lets suppose I going to spend some time for testing. How would I 
> install glusterfs-server including your feature?
>
> Maybe this is an easy procedure, but actually I am not familiar with it.
>
> Regards,
>
> Felix
>
> On 27/05/2020 07:56, RAFI KC wrote:
>>
>> Hi All,
>>
>> I have been working on POC to improve readdirp performance 
>> improvement. At the end of the experiment, The results are showing 
>> promising result in performance, overall there is a 104% improvement 
>> for full filesystem crawl compared to the existing solution. Here is 
>> the short test numbers. The tests were carried out in 16*3 setup with 
>> 1.5 Million dentries (Both files and dir). The system also contains 
>> some empty directories. *In the result the proposed solution is 287% 
>> faster than the plane volume and 104% faster than the 
>> parallel-readdir based solution.*
>>
>> *
>> *
>>
>> Configuration
>>
>> 	
>>
>> Plain volume
>>
>> 	
>>
>> Parallel-readdir
>>
>> 	
>>
>> Proposed Solution
>>
>> FS Crawl Time in Seconds
>>
>> 	
>>
>> 16497.523
>>
>> 	
>>
>> 8717.872
>>
>> 	
>>
>> 4261.401
>>
>> In short, the basic idea behind the proposal is the efficient 
>> managing of readdir buffer in gluster along with prefetching the 
>> dentries for intelligent switch-over to the next buffer. The detailed 
>> problem description, deign description and results are available in 
>> the 
>> doc.https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit 
>>
>>
>>
>> If anybody can help with the testing on a different kind of 
>> workloads, I would be very happy to assist. If wanted to test the 
>> patch and run a performance test on your setup, I could help with 
>> back-porting the patch to the version of your choice.
>>
>>
>> https://review.gluster.org/24469
>>
>> https://review.gluster.org/24470
>>
>>
>> Regards
>>
>> Rafi KC
>>
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge:https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200527/6317edc5/attachment.html>