[Bugs] [Bug 1478411] Directory listings on fuse mount are very slow due to small number of getdents () entries

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1478411

Raghavendra G <rgowdapp at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
              Flags|needinfo?(rgowdapp at redhat.c |needinfo?(nh2-redhatbugzill
                   |om)                         |a at deditus.de)



--- Comment #36 from Raghavendra G <rgowdapp at redhat.com> ---
> I have another question about what you said:

> You mentioned that the "translator readdir-ahead tries to hide the readdirplus latency on bricks". Is readdir-latency on bricks really an issue?
For example, `ls -1U /data/glusterfs/myvol/brick1/brick/largenames-100k/` takes
only 0.15 seconds real time on the brick when the directory is currently being
written to, and 0.06 seconds if it's not being written to.
> On the `largenames-10k` on the brick, it's another 10x faster than for 100k, as expected.
> An individual `getdents(3, /* 455 entries */, 32768) = 32760` call on the brick takes between 40 and 400 micro-seconds (average 80 us) on the brick when the directory is being written to, and 4x less when it's not being written to.
> So I would find it very surprising if readdir latency on the brick FS itself could make any noticeable impact; can you explain it a bit?

Glusterfs does following things on a brick during a readdirplus call (A
readdirplus - like NFS readdirplus - is a readdir + collection of metadata like
xattrs, stat for each dentry so that kernel don't have to do a lookup/stat for
each dentry and by default every readdir is converted to readdirplus):
* getdents
* For each dentry returned by getdents do:
  * lstat (path pointed out by dentry)
  * multiple getxattr calls to fetch various xattrs storing glusterfs metadata
on the file.

So, there is list processing involved here. Note also that this list processing
happens more than once at different layers. This list processing could be the
reason for cpu usage.
Note that we can try some more experiments:

Experiment 1 - Isolate the overhead required to access metadata of each dentry:
===============================================================================

* Disable readdirplus entirely in glusterfs stack. Refer to [1] for details on
how to do it.
* Run a crawler like find or ls, but without accessing each dentry itself. IOW,
we just list the contents of the directory. This will help us to find out how
much of overhead is spent in collecting metadata of contents of directory.

Experiment 2 - Isolate the overhead of iterating through list of dentries:
==========================================================================

In this experiment we'll fetch metadata of each dentry, but without using
readdirplus. As I mentioned above, different xlators traverse through the list
in the scope of a single readdirplus call. So, there are multiple iterations.
However, we can still access metadata of each dentry without using readdirplus.
In this case kernel initiates lookup/stat calls for each dentry it fetched
through readdir. The crucial difference is that list processing is minimal in
glusterfs as most of the xlators are not interested in just dentries (instead
they are interested in inode pointed out by dentry. Since readdir doesn't have
inode/stat information they just pass the list to higher layers without
processing it).

* Disable readdirplus entirely in glusterfs stack. Refer to [1] for details on
how to do it.
* Run a crawler like find or ls, but this time let it access each file. We can
use commands like "ls -lR" or find . -exec stat \{} \;

We can compare results from these two experiments with the information provided
in comment #35.

[1] http://lists.gluster.org/pipermail/gluster-users/2017-March/030148.html

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.