[Gluster-users] It appears that readdir is not cached for FUSE mounts
hunter86_bg at yahoo.com
Mon Feb 10 16:31:13 UTC 2020
On February 10, 2020 5:32:29 PM GMT+02:00, Matthias Schniedermeyer <matthias-gluster-users at maxcluster.de> wrote:
>On 10.02.20 16:21, Strahil Nikolov wrote:
>> On February 10, 2020 2:25:17 PM GMT+02:00, Matthias Schniedermeyer
><matthias-gluster-users at maxcluster.de> wrote:
>>> I would describe our basic use case for gluster as:
>>> "data-store for a cold-standby application".
>>> A specific application is installed on 2 hardware machines, the data
>>> kept in-sync between the 2 machines by a replica-2 gluster volume.
>>> (IOW: "RAID 1")
>>> At any one time only 1 machine has the volume mounted and the
>>> application running. If the machine goes down the application is
>>> on the remaining machine.
>>> IOW at any one point in time there is only ever 1 "reader & writer"
>>> I profiled a performance problem we have with this application,
>>> unfortunately we can't modify.
>>> The profile shows many "opendir/readdirp/releasedir" cycles, the
>>> directory in question has about 1000 files and the application
>>> for several milliseconds any time it decides to do a readdir.
>>> The volume is mounted via FUSE and it appears that said operation is
>>> cached at all.
>>> To provide a test-case i tried to replicate what the application
>>> The problematic operation is nearly perfectly emulated just by using
>>> "ls .".
>>> I created a script that replicates how we use gluster and
>>> that a FUSE-mount appears to be lacking any caching of readdir.
>>> A word about the test-environment:
>>> 2 identical servers
>>> Dual Socket Xeon CPU E5-2640 v3 (8 cores, 2.60GHz, HT enabled)
>>> RAM: 128GB DDR4 ECC (8x16GB)
>>> Storage: 2TB Intel P3520 PCIe-NVMe-SSD
>>> Network: Gluster: 10GB/s direct connect (no switch), external:
>>> OS: CentOS 7.7, Installed with "Minimal" ISO, everything: Default
>>> Up2Date as of: 2020-01-21 (Kernel: 3.10.0-1062.9.1.el7.x86_64)
>>> SELinux: Disabled
>>> SSH-Key for 1 -> 2 exchanged
>>> Gluster 6.7 packages installed via 'centos-release-gluster6'
>>> see attached:
>>> The meat of the testcase is this:
>>> a profile of:
>>> ls .
>>> ls . . . . . . . . . .
>>> (10 dots)
>>>> cat /root/profile-1-times | grep DIR | head -n 3
>>> 0.00 0.00 us 0.00 us 0.00 us 1
>>> 0.27 66.79 us 66.79 us 66.79 us 1
>>> 98.65 12190.30 us 9390.88 us 14989.73 us 2
>>>> cat /root/profile-10-times | grep DIR | head -n 3
>>> 0.00 0.00 us 0.00 us 0.00 us 10
>>> 0.64 108.02 us 85.72 us 131.96 us 10
>>> 99.36 8388.64 us 5174.71 us 14808.77 us 20
>>> This testcase shows perfect scaling.
>>> 10 times the request, results in 10 times the gluster-operations.
>>> I would say ideally there should be no difference in the number of
>>> gluster-operations, regardless of how often a directory is read in a
>>> short amount of time (with no changes in between)
>>> Is there something we can do to enable caching or otherwise improve
>> Hi Matthias,
>> Have you tried the 'readdir-ahead' option .
>> According to docs it is useful for ' improving sequential directory
>read performance' .
>> I'm not sure how gluster defines sequential directory read, but it's
>readdir-ahead is enabled by default. Has been for several years.
>In effect this option changes how many READDIRP OPs are executed for a
>single "ls .".
>(It takes more OPs when readdir-ahead is disabled.)
>> Also, you can try metadata caching , as described in:
>> The actual group should contain the following:
>Metadata-Caching, in general, works:
>e.g. `stat FILE` is cached if executed repeatedly.
>AFAICT the big exception to metadata-caching is readdir.
This now has turned into 'shoot into the dark'.
I have checked a nice presentation and these 2 attracted my attention:
Presentation is found at:
I hope you find something useful there.
More information about the Gluster-users