[Gluster-users] It appears that readdir is not cached for FUSE mounts

Wed Feb 19 11:22:19 UTC 2020

On 10.02.20 17:31, Strahil Nikolov wrote:
> On February 10, 2020 5:32:29 PM GMT+02:00, Matthias Schniedermeyer <matthias-gluster-users at maxcluster.de> wrote:
>> On 10.02.20 16:21, Strahil Nikolov wrote:
>>> On February 10, 2020 2:25:17 PM GMT+02:00, Matthias Schniedermeyer
>> <matthias-gluster-users at maxcluster.de> wrote:
>>>> Hi
>>>>
>>>>
>>>> I would describe our basic use case for gluster as:
>>>> "data-store for a cold-standby application".
>>>>
>>>> A specific application is installed on 2 hardware machines, the data
>> is
>>>> kept in-sync between the 2 machines by a replica-2 gluster volume.
>>>> (IOW: "RAID 1")
>>>>
>>>> At any one time only 1 machine has the volume mounted and the
>>>> application running. If the machine goes down the application is
>>>> started
>>>> on the remaining machine.
>>>> IOW at any one point in time there is only ever 1 "reader & writer"
>>>> running.
>>>>
>>>> I profiled a performance problem we have with this application,
>> which
>>>> unfortunately we can't modify.
>>>>
>>>> The profile shows many "opendir/readdirp/releasedir" cycles, the
>>>> directory in question has about 1000 files and the application
>> "stalls"
>>>> for several milliseconds any time it decides to do a readdir.
>>>> The volume is mounted via FUSE and it appears that said operation is
>>>> not
>>>> cached at all.
>>>>
>>>> To provide a test-case i tried to replicate what the application
>> does.
>>>> The problematic operation is nearly perfectly emulated just by using
>>>> "ls .".
>>>>
>>>> I created a script that replicates how we use gluster and
>> demonstrates
>>>> that a FUSE-mount appears to be lacking any caching of readdir.
>>>>
>>>> A word about the test-environment:
>>>> 2 identical servers
>>>> Dual Socket Xeon CPU E5-2640 v3 (8 cores, 2.60GHz, HT enabled)
>>>> RAM: 128GB DDR4 ECC (8x16GB)
>>>> Storage: 2TB Intel P3520 PCIe-NVMe-SSD
>>>> Network: Gluster: 10GB/s direct connect (no switch), external:
>> 1Gbit/s
>>>> OS: CentOS 7.7, Installed with "Minimal" ISO, everything: Default
>>>> Up2Date as of: 2020-01-21 (Kernel: 3.10.0-1062.9.1.el7.x86_64)
>>>> SELinux: Disabled
>>>> SSH-Key for 1 -> 2 exchanged
>>>> Gluster 6.7 packages installed via 'centos-release-gluster6'
>>>>
>>>> see attached:
>> gluster-testcase-no-caching-of-dir-operations-for-fuse.sh
>>>>
>>>> The meat of the testcase is this:
>>>> a profile of:
>>>> ls .
>>>> vs:
>>>> ls . . . . . . . . . .
>>>> (10 dots)
>>>>
>>>>> cat /root/profile-1-times | grep DIR | head -n 3
>>>> 0.00       0.00 us       0.00 us       0.00 us              1
>>>> RELEASEDIR
>>>> 0.27      66.79 us      66.79 us      66.79 us              1
>> OPENDIR
>>>> 98.65   12190.30 us    9390.88 us   14989.73 us              2
>>>> READDIRP
>>>>
>>>>> cat /root/profile-10-times | grep DIR | head -n 3
>>>> 0.00       0.00 us       0.00 us       0.00 us             10
>>>> RELEASEDIR
>>>> 0.64     108.02 us      85.72 us     131.96 us             10
>> OPENDIR
>>>> 99.36    8388.64 us    5174.71 us   14808.77 us             20
>>>> READDIRP
>>>>
>>>> This testcase shows perfect scaling.
>>>> 10 times the request, results in 10 times the gluster-operations.
>>>>
>>>> I would say ideally there should be no difference in the number of
>>>> gluster-operations, regardless of how often a directory is read in a
>>>> short amount of time (with no changes in between)
>>>>
>>>>
>>>> Is there something we can do to enable caching or otherwise improve
>>>> performance?
>>>
>>> Hi Matthias,
>>>
>>> Have you tried the 'readdir-ahead' option .
>>> According to docs it is useful for ' improving sequential directory
>> read performance' .
>>> I'm not sure how gluster defines sequential directory read, but it's
>> worth trying.
>>
>> readdir-ahead is enabled by default. Has been for several years.
>> In effect this option changes how many READDIRP OPs are executed for a
>> single "ls .".
>> (It takes more OPs when readdir-ahead is disabled.)
>>
>>> Also, you can try metadata caching , as described in:
>>>
>> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/sect-directory_operations
>>> The actual group should contain the following:
>>>
>> https://github.com/gluster/glusterfs/blob/master/extras/group-metadata-cache
>>
>> Metadata-Caching, in general, works:
>> e.g. `stat FILE` is cached if executed repeatedly.
>>
>> AFAICT the big exception to metadata-caching is readdir.
> 
> Hi Matthias,
> 
> This now has turned into 'shoot into the dark'.
> 
> I have checked a nice presentation and these 2 attracted my attention:
> performance.parallel-readdir on
> cluster.readdir-hashed on
> 
> Presentation is found at:
> https://www.google.com/url?sa=t&source=web&rct=j&url=https://events.static.linuxfound.org/sites/events/files/slides/Gluster_DirPerf_Vault2017_0.pdf&ved=2ahUKEwirh-qBs8fnAhWTTxUIHfn3CWEQFjAAegQIAhAB&usg=AOvVaw1yhHZaWovhYGCexkGaMVQ8&cshid=1581352097024
> 
> I hope you find something useful there.

Unfortunatly: No
I tried the options, but they have "no effect".

 From the names they both sound to me like they are for distributed volumes. Where you have to piece together several (partial) readdirs, to generate a whole.
In my case the volume is replicated and the contents of any dir in any brick is the same (given a healthy & up2date status).

I guess my "problem" is less in the gluster-parts, but more in the fuse-part of the equation.

-- 
Matthias