[Gluster-users] Readdirp (ls -l) Performance Improvement

[RAFI KC]

Sure, I have back-ported the patch to release-7. Now I will see How I 
can build the rpms.

On the other hand, if possible, Could you please tell me about the 
volume configuration and the volume tree structure. This will be more 
helpful when you have lot of files or a distribution is high.

Regards

Rafi KC

On 27/05/20 5:49 pm, Strahil Nikolov wrote:
> Hi Rafi,
>
> I have a test oVirt 4.3.9 cluster  with Gluster v7.5 on CentOS7.
> Can you provide the rpms and I will try to test.
>
> Also, please share the switch that disables this behaviour (in case something goes wrong).
>
> Best Regards,
> Strahil Nikolov
>
> На 27 май 2020 г. 14:54:34 GMT+03:00, RAFI KC <rkavunga at redhat.com> написа:
>> Hi Felix,
>>
>> If you already have a test cluster, please give me the gluster version.
>>
>> I will try to build rpm then you can just do an upgrade. If you don't
>> have a test cluster, and you are creating a new one, then I will build
>> a
>> new rpm against the latest head. This is more easier to me as I don't
>> have to backport to a previous version.
>>
>> May I ask you the access protocol that you use (FUSE, SMB , NFS,
>> gfapi).
>> I did the testing on a fuse protocol.
>>
>>
>> Rafi KC
>>
>> On 27/05/20 4:14 pm, Felix Kölzow wrote:
>>> Dear Rafi KC,
>>>
>>>
>>> lets suppose I going to spend some time for testing. How would I
>>> install glusterfs-server including your feature?
>>>
>>> Maybe this is an easy procedure, but actually I am not familiar with
>> it.
>>> Regards,
>>>
>>> Felix
>>>
>>> On 27/05/2020 07:56, RAFI KC wrote:
>>>> Hi All,
>>>>
>>>> I have been working on POC to improve readdirp performance
>>>> improvement. At the end of the experiment, The results are showing
>>>> promising result in performance, overall there is a 104% improvement
>>>> for full filesystem crawl compared to the existing solution. Here is
>>>> the short test numbers. The tests were carried out in 16*3 setup
>> with
>>>> 1.5 Million dentries (Both files and dir). The system also contains
>>>> some empty directories. *In the result the proposed solution is 287%
>>>> faster than the plane volume and 104% faster than the
>>>> parallel-readdir based solution.*
>>>>
>>>> *
>>>> *
>>>>
>>>> Configuration
>>>>
>>>> 	
>>>>
>>>> Plain volume
>>>>
>>>> 	
>>>>
>>>> Parallel-readdir
>>>>
>>>> 	
>>>>
>>>> Proposed Solution
>>>>
>>>> FS Crawl Time in Seconds
>>>>
>>>> 	
>>>>
>>>> 16497.523
>>>>
>>>> 	
>>>>
>>>> 8717.872
>>>>
>>>> 	
>>>>
>>>> 4261.401
>>>>
>>>> In short, the basic idea behind the proposal is the efficient
>>>> managing of readdir buffer in gluster along with prefetching the
>>>> dentries for intelligent switch-over to the next buffer. The
>> detailed
>>>> problem description, deign description and results are available in
>>>> the
>>>>
>> doc.https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit
>>
>>>>
>>>>
>>>> If anybody can help with the testing on a different kind of
>>>> workloads, I would be very happy to assist. If wanted to test the
>>>> patch and run a performance test on your setup, I could help with
>>>> back-porting the patch to the version of your choice.
>>>>
>>>>
>>>> https://review.gluster.org/24469
>>>>
>>>> https://review.gluster.org/24470
>>>>
>>>>
>>>> Regards
>>>>
>>>> Rafi KC
>>>>
>>>>
>>>> ________
>>>>
>>>>
>>>>
>>>> Community Meeting Calendar:
>>>>
>>>> Schedule -
>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>> Bridge:https://bluejeans.com/441850968
>>>>
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users