[Gluster-devel] Fwd: Change in glusterfs[master]: performance/readdir-ahead: translator prototype

[John Mark Walker]

Interesting development work here, if anyone is interested.

-JM


----- Forwarded Message -----
From: "Brian Foster (Code Review)" <root at dev.gluster.com>
Sent: Friday, February 15, 2013 9:21:41 AM
Subject: Change in glusterfs[master]: performance/readdir-ahead: translator prototype

Brian Foster has uploaded a new change for review.

Change subject: performance/readdir-ahead: translator prototype
......................................................................

performance/readdir-ahead: translator prototype

This is a prototype translator to explore the potential performance
benefits of directory prefetching on sequential directory reads. It
is incomplete and only tested with a focus on performance of the
specific use case of single-threaded, sequential directory reads.
General thoughts on approach/design are appreciated.

While directory read performance could still be considered slow with
this functionality, preliminary results show a measurable
improvement:

Notes:
- The test directory structures are one or two levels deep. 'ls -R'
  is used to ensure lookups are generated during the traversal.
- The biggest issue in performance of gluster readdirp tests is the
  ineffectiveness of md-cache during the initial traversal. Inodes
  are not linked until the completion of the first pass, causing
  lookups to dominate the test and increase the time to complete by
  orders of magnitude. This is a separate issue with fixes in
  progress. Thus, all tests that follow are run after an initial
  'ls -R' to ensure inodes are linked into the inode table and
  md-cache is working.

Test Description:
- 2-brick DHT volume, all translators other than md-cache disabled.
- gid-timeout=2 on client mount.
- Command: time ls -R /mnt/<largedir> | wc -l

Results:

--- 100k files (1 dir)

- baseline - ~13s
- readdir-ahead - ~6s

--- 200k files (2 dirs, 100k per dir)

- baseline - ~28s
- readdir-ahead - ~12s

--- 1m files (10 dirs, 100k per dir)

- baseline - 2m32s
- readdir-ahead - 1m13s

Change-Id: Ieceb9e1eb47d1d5b5af8da2bf03839537364653f
Signed-off-by: Brian Foster <bfoster at redhat.com>
---
M configure.ac
M xlators/performance/Makefile.am
A xlators/performance/readdir-ahead/Makefile.am
A xlators/performance/readdir-ahead/src/Makefile.am
A xlators/performance/readdir-ahead/src/readdir-ahead-mem-types.h
A xlators/performance/readdir-ahead/src/readdir-ahead.c
A xlators/performance/readdir-ahead/src/readdir-ahead.h
7 files changed, 552 insertions(+), 1 deletion(-)


  git pull ssh://git.gluster.org/glusterfs refs/changes/19/4519/1
--
To view, visit http://review.gluster.org/4519
To unsubscribe, visit http://review.gluster.org/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Ieceb9e1eb47d1d5b5af8da2bf03839537364653f
Gerrit-PatchSet: 1
Gerrit-Project: glusterfs
Gerrit-Branch: master
Gerrit-Owner: Brian Foster <bfoster at redhat.com>