[Gluster-users] Readdirp (ls -l) Performance Improvement

[Strahil Nikolov]

Hey Rafi,

what do you mean with volume configuration and tree structure.

Best Regards,
Strahil Nikolov

На 27 май 2020 г. 16:18:36 GMT+03:00, RAFI KC <rkavunga at redhat.com> написа:
>Sure, I have back-ported the patch to release-7. Now I will see How I 
>can build the rpms.
>
>On the other hand, if possible, Could you please tell me about the 
>volume configuration and the volume tree structure. This will be more 
>helpful when you have lot of files or a distribution is high.
>
>Regards
>
>Rafi KC
>
>On 27/05/20 5:49 pm, Strahil Nikolov wrote:
>> Hi Rafi,
>>
>> I have a test oVirt 4.3.9 cluster  with Gluster v7.5 on CentOS7.
>> Can you provide the rpms and I will try to test.
>>
>> Also, please share the switch that disables this behaviour (in case
>something goes wrong).
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> На 27 май 2020 г. 14:54:34 GMT+03:00, RAFI KC <rkavunga at redhat.com>
>написа:
>>> Hi Felix,
>>>
>>> If you already have a test cluster, please give me the gluster
>version.
>>>
>>> I will try to build rpm then you can just do an upgrade. If you
>don't
>>> have a test cluster, and you are creating a new one, then I will
>build
>>> a
>>> new rpm against the latest head. This is more easier to me as I
>don't
>>> have to backport to a previous version.
>>>
>>> May I ask you the access protocol that you use (FUSE, SMB , NFS,
>>> gfapi).
>>> I did the testing on a fuse protocol.
>>>
>>>
>>> Rafi KC
>>>
>>> On 27/05/20 4:14 pm, Felix Kölzow wrote:
>>>> Dear Rafi KC,
>>>>
>>>>
>>>> lets suppose I going to spend some time for testing. How would I
>>>> install glusterfs-server including your feature?
>>>>
>>>> Maybe this is an easy procedure, but actually I am not familiar
>with
>>> it.
>>>> Regards,
>>>>
>>>> Felix
>>>>
>>>> On 27/05/2020 07:56, RAFI KC wrote:
>>>>> Hi All,
>>>>>
>>>>> I have been working on POC to improve readdirp performance
>>>>> improvement. At the end of the experiment, The results are showing
>>>>> promising result in performance, overall there is a 104%
>improvement
>>>>> for full filesystem crawl compared to the existing solution. Here
>is
>>>>> the short test numbers. The tests were carried out in 16*3 setup
>>> with
>>>>> 1.5 Million dentries (Both files and dir). The system also
>contains
>>>>> some empty directories. *In the result the proposed solution is
>287%
>>>>> faster than the plane volume and 104% faster than the
>>>>> parallel-readdir based solution.*
>>>>>
>>>>> *
>>>>> *
>>>>>
>>>>> Configuration
>>>>>
>>>>> 	
>>>>>
>>>>> Plain volume
>>>>>
>>>>> 	
>>>>>
>>>>> Parallel-readdir
>>>>>
>>>>> 	
>>>>>
>>>>> Proposed Solution
>>>>>
>>>>> FS Crawl Time in Seconds
>>>>>
>>>>> 	
>>>>>
>>>>> 16497.523
>>>>>
>>>>> 	
>>>>>
>>>>> 8717.872
>>>>>
>>>>> 	
>>>>>
>>>>> 4261.401
>>>>>
>>>>> In short, the basic idea behind the proposal is the efficient
>>>>> managing of readdir buffer in gluster along with prefetching the
>>>>> dentries for intelligent switch-over to the next buffer. The
>>> detailed
>>>>> problem description, deign description and results are available
>in
>>>>> the
>>>>>
>>>
>doc.https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit
>>>
>>>>>
>>>>>
>>>>> If anybody can help with the testing on a different kind of
>>>>> workloads, I would be very happy to assist. If wanted to test the
>>>>> patch and run a performance test on your setup, I could help with
>>>>> back-porting the patch to the version of your choice.
>>>>>
>>>>>
>>>>> https://review.gluster.org/24469
>>>>>
>>>>> https://review.gluster.org/24470
>>>>>
>>>>>
>>>>> Regards
>>>>>
>>>>> Rafi KC
>>>>>
>>>>>
>>>>> ________
>>>>>
>>>>>
>>>>>
>>>>> Community Meeting Calendar:
>>>>>
>>>>> Schedule -
>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>> Bridge:https://bluejeans.com/441850968
>>>>>
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>> ________
>>>>
>>>>
>>>>
>>>> Community Meeting Calendar:
>>>>
>>>> Schedule -
>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>> Bridge: https://bluejeans.com/441850968
>>>>
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users