[Gluster-users] Small File and "ls" performance ..

[Joshua Baker-LePain]

On Wed, 5 May 2010 at 1:00am, Tejas N. Bhise wrote

> We have recently made some code changes in an effort to improve small
> file and 'ls' performance. The patches are -
>
> selective readdirp - http://patches.gluster.com/patch/3203/
> dht lookup revalidation optimization - http://patches.gluster.com/patch/3204/
> updated write-behind default values - http://patches.gluster.com/patch/3223/
>
> DISCLAIMER : These patches have not made it to any supported release
> yet and have not been tested yet. Don't use them in production. I am
> providing this information only as some advance notice for those in
> the community who might be interested in trying out these changes and
> provide feedback.

I would like to cast my vote firmly in favor of these patches.  I did a 
bit of a torture test with my scratch gluster setup:

Storage bricks:
10 HP DL160 G5s, each with a single 7200RPM SATA disk

Client:
Same hardware.  Everything is connected via GbE to the same switch.

Gluster setup:
Gluster 3.0.4.  Standard replicate then distribute setup created via 
gluster-volgen.

Test:
An old version of <http://people.redhat.com/dledford/memtest.shtml>. 
This script unpacks *lots* of copies of the Linux kernel tarball (it's 
based on memory size -- for this client, it was 98), diffs all of them 
against the first copy, and then removes them all.  So, lots of small 
files.

Length of 1 run before patches:
5622m56.020s

Length of 1 run after patches:
711m54.006s

Wow.  And the run with the test patches didn't generate any errors.

> Once these are fully tested they will make to an officially supported release.

I rather look forward to that.

-- 
Joshua Baker-LePain
QB3 Shared Cluster Sysadmin
UCSF