[Bugs] [Bug 1478411] Directory listings on fuse mount are very slow due to small number of getdents () entries

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1478411



--- Comment #37 from Raghavendra G <rgowdapp at redhat.com> ---
(In reply to Raghavendra G from comment #36)
> > I have another question about what you said:
> 
> > You mentioned that the "translator readdir-ahead tries to hide the readdirplus latency on bricks". Is readdir-latency on bricks really an issue?
> For example, `ls -1U /data/glusterfs/myvol/brick1/brick/largenames-100k/`
> takes only 0.15 seconds real time on the brick when the directory is
> currently being written to, and 0.06 seconds if it's not being written to.
> > On the `largenames-10k` on the brick, it's another 10x faster than for 100k, as expected.
> > An individual `getdents(3, /* 455 entries */, 32768) = 32760` call on the brick takes between 40 and 400 micro-seconds (average 80 us) on the brick when the directory is being written to, and 4x less when it's not being written to.
> > So I would find it very surprising if readdir latency on the brick FS itself could make any noticeable impact; can you explain it a bit?
> 
> Glusterfs does following things on a brick during a readdirplus call (A
> readdirplus - like NFS readdirplus - is a readdir + collection of metadata
> like xattrs, stat for each dentry so that kernel don't have to do a
> lookup/stat for each dentry and by default every readdir is converted to
> readdirplus):
> * getdents
> * For each dentry returned by getdents do:
>   * lstat (path pointed out by dentry)
>   * multiple getxattr calls to fetch various xattrs storing glusterfs
> metadata on the file.
> 
> So, there is list processing involved here. Note also that this list
> processing happens more than once at different layers. This list processing
> could be the reason for cpu usage.
> Note that we can try some more experiments:
> 
> Experiment 1 - Isolate the overhead required to access metadata of each
> dentry:
> =============================================================================
> ==
> 
> * Disable readdirplus entirely in glusterfs stack. Refer to [1] for details
> on how to do it.
> * Run a crawler like find or ls, but without accessing each dentry itself.
> IOW, we just list the contents of the directory. This will help us to find
> out how much of overhead is spent in collecting metadata of contents of
> directory.

We can use "find . > /dev/null" to do this.

> 
> Experiment 2 - Isolate the overhead of iterating through list of dentries:
> ==========================================================================
> 
> In this experiment we'll fetch metadata of each dentry, but without using
> readdirplus. As I mentioned above, different xlators traverse through the
> list in the scope of a single readdirplus call. So, there are multiple
> iterations. However, we can still access metadata of each dentry without
> using readdirplus. In this case kernel initiates lookup/stat calls for each
> dentry it fetched through readdir. The crucial difference is that list
> processing is minimal in glusterfs as most of the xlators are not interested
> in just dentries (instead they are interested in inode pointed out by
> dentry. Since readdir doesn't have inode/stat information they just pass the
> list to higher layers without processing it).
> 
> * Disable readdirplus entirely in glusterfs stack. Refer to [1] for details
> on how to do it.
> * Run a crawler like find or ls, but this time let it access each file. We
> can use commands like "ls -lR" or find . -exec stat \{} \;
> 
> We can compare results from these two experiments with the information
> provided in comment #35.
> 
> [1] http://lists.gluster.org/pipermail/gluster-users/2017-March/030148.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.