[Gluster-users] Readdirp (ls -l) Performance Improvement

[Strahil Nikolov]

Hi Rafi,

I have a test oVirt 4.3.9 cluster  with Gluster v7.5 on CentOS7.
Can you provide the rpms and I will try to test.

Also, please share the switch that disables this behaviour (in case something goes wrong).

Best Regards,
Strahil Nikolov

На 27 май 2020 г. 14:54:34 GMT+03:00, RAFI KC <rkavunga at redhat.com> написа:
>Hi Felix,
>
>If you already have a test cluster, please give me the gluster version.
>
>I will try to build rpm then you can just do an upgrade. If you don't 
>have a test cluster, and you are creating a new one, then I will build
>a 
>new rpm against the latest head. This is more easier to me as I don't 
>have to backport to a previous version.
>
>May I ask you the access protocol that you use (FUSE, SMB , NFS,
>gfapi). 
>I did the testing on a fuse protocol.
>
>
>Rafi KC
>
>On 27/05/20 4:14 pm, Felix Kölzow wrote:
>>
>> Dear Rafi KC,
>>
>>
>> lets suppose I going to spend some time for testing. How would I 
>> install glusterfs-server including your feature?
>>
>> Maybe this is an easy procedure, but actually I am not familiar with
>it.
>>
>> Regards,
>>
>> Felix
>>
>> On 27/05/2020 07:56, RAFI KC wrote:
>>>
>>> Hi All,
>>>
>>> I have been working on POC to improve readdirp performance 
>>> improvement. At the end of the experiment, The results are showing 
>>> promising result in performance, overall there is a 104% improvement
>
>>> for full filesystem crawl compared to the existing solution. Here is
>
>>> the short test numbers. The tests were carried out in 16*3 setup
>with 
>>> 1.5 Million dentries (Both files and dir). The system also contains 
>>> some empty directories. *In the result the proposed solution is 287%
>
>>> faster than the plane volume and 104% faster than the 
>>> parallel-readdir based solution.*
>>>
>>> *
>>> *
>>>
>>> Configuration
>>>
>>> 	
>>>
>>> Plain volume
>>>
>>> 	
>>>
>>> Parallel-readdir
>>>
>>> 	
>>>
>>> Proposed Solution
>>>
>>> FS Crawl Time in Seconds
>>>
>>> 	
>>>
>>> 16497.523
>>>
>>> 	
>>>
>>> 8717.872
>>>
>>> 	
>>>
>>> 4261.401
>>>
>>> In short, the basic idea behind the proposal is the efficient 
>>> managing of readdir buffer in gluster along with prefetching the 
>>> dentries for intelligent switch-over to the next buffer. The
>detailed 
>>> problem description, deign description and results are available in 
>>> the 
>>>
>doc.https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit
>
>>>
>>>
>>>
>>> If anybody can help with the testing on a different kind of 
>>> workloads, I would be very happy to assist. If wanted to test the 
>>> patch and run a performance test on your setup, I could help with 
>>> back-porting the patch to the version of your choice.
>>>
>>>
>>> https://review.gluster.org/24469
>>>
>>> https://review.gluster.org/24470
>>>
>>>
>>> Regards
>>>
>>> Rafi KC
>>>
>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge:https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users