[Bugs] [Bug 1478411] Directory listings on fuse mount are very slow due to small number of getdents () entries

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1478411

Poornima G <pgurusid at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |pgurusid at redhat.com
              Flags|                            |needinfo?(nh2-redhatbugzill
                   |                            |a at deditus.de)



--- Comment #7 from Poornima G <pgurusid at redhat.com> ---
(In reply to nh2 from comment #6)
> This bug is NOT fixed with Gluster 3.12, I just tried it:
> 
> $ gluster --version
> glusterfs 3.12.0
> 
> $ for x in `seq 1 1000`; do echo $x; touch /myglustermountdir/$x; done
> 
> $ strace find /myglustermountdir
> ...
> getdents(4, /* 24 entries */, 131072)   = 576
> getdents(4, /* 23 entries */, 131072)   = 552
> getdents(4, /* 23 entries */, 131072)   = 552
> getdents(4, /* 23 entries */, 131072)   = 552
> ...


Thanks for the data. So here is my analysis and possible solution:

1. You are right, even after the BZ bug 1356453 is fixed, the getdents only
fetches 20-26 entries.

2. The reason for the same is FUSE kernel module behaviour:
   FUSE, irrespective of what buffer getdents sends, fuse limits it to
PAGE_SIZE 4k(4096) and then sends it to gluster. Thus because of fuse, gluster
getdent buffer is limited to 4k. But when you assign the gedents buffer size to
4K and run the same on plain XFS, there will be ~128 entries returned, which is
much higher than 20-26 returned by FUSE/Gluster. The reason for this is, the
structure fuse uses to getdents is (struct fuse_direntplus) and the buffer size
mentioned in application is for (struct linux_dirent). The sizeof(struct
fuse_direntplus) is ~158 and sizeof(struct linux_dirent) is ~24, thus the
number od dentries that can be accommodated by fuse getdents buffer in 4K is ~5
times less than that can be accommodated by application getdents.

   Thus we are limited by the FUSE buffer size and struture type.

3. But the saving grace is, each getdent call is not a network round trip like
you mentioned in initial comment. This is because of readdir-ahead caching in
gluster. Gluster basically pre-fetches the dentries and keeps it in cache. The
pre-fetching buf size is 128K and cache buffer size is 10MB default. This 99%
of the getdents gets served from this cache and doesn't result in network round
trip. So, even if FUSE fixes getdents buffer thing, you may not see 30X
improvement as you expected, but some percentage only, as it saves lots of
getdents call and kernel to user context switches.


I would suggest we close this bug on gluster, and raise it in FUSE kernel?

An alternative for this, we can implement glfs_getdents in libgfapi(system call
equivalent of GlusterFS and get rid of the FUSE interference) which can be
integrated to other applications or we can write a wrapper and provide it as a
command.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.