[Gluster-users] very low file creation rate with glusterfs

Wei Dong wdong.pku at gmail.com
Thu Sep 10 18:05:58 UTC 2009

Hi All,

I complained about the low file creation rate with the glusterfs on my
cluster weeks ago and Avati suggested I started with a small number of
nodes.  I finally get sometime to seriously benchmark glusterfs with
Bonnie++ today and the results confirms that glusterfs is indeed slow in
terms of file creating.  My application is to store a large number of ~200KB
image files.  I use the following bonnie++ command for evaluation (create
10K files of 200KiB each scattered under 100 directories):

bonnie++ -d . -s 0 -n 10:200000:200000:100

Since sequential I/O is not that interesting to me, I only keep the random
I/O results.

My hardware configuration is 2xquadcore Xeon E5430 2.66GHz, 16GB memory, 4 x
Seagate 1500GiB 7200RPM hard drive.  The machines are connected with gigabit

I ran several GlusterFS configurations, each named as N-R-T, where N is the
number of replicated volumes aggregated, R is the number of replications and
T is number of server side I/O thread.  I use one machine to serve one
volume so there are NxR servers and one separate client running for each
experiment.  On the client side, the server volumes are first replicated and
then aggregated -- even with 1-1-2 configuration, the single volume is
wrapped by a replicate and a distribute translator.  To show the overhead of
those translators, I also run a "simple" configuration which is 1-1-2
without the extra replicate & distribute translators, and a "local"
configuration which is "simple" with client & server running on the same
machine.  These configurations are compared to "nfs" and "nfs-local", which
is NFS with server and client on the same machine.  The GlusterFS volume
file templates are attached to the email.

The result is at
 The bars/numbers shown are operations/second, so the larger the

Following are the messages shown by the figure:
1.  GlusterFS is doing a exceptionally good job on deleting files, but
creates and reads files much slower than both NFS.
2.  At least for one node server configuration, network doesn't affects the
file creation rate and does affects file read rate.
3.  The extra dummy replicate & distribute translators lowers file creation
rate by almost half. 4.  Replication doesn't hurt performance a lot.
5.  I'm running only single-threaded benchmark, so it's hard to say about
scalability, but adding more servers does helps a little bit even in
single-threaded setting.

Note that my results are not really that different from
where the single node configuration file create rate is about 30/second.

I see no reason why GlusterFS has to be that slower than NFS in file
creation in single node configuration.  I'm wondering if someone here can
help me figure out what's wrong in my configuration or what's wrong in the
GlusterFS implementation.

- Wei

Server volume:

volume posix
 type storage/posix
 option directory /state/partition1/wdong/gluster

volume lock
 type features/locks
 subvolumes posix

volume brick
 type performance/io-threads
 option thread-count 2
 subvolumes lock

volume server
 type protocol/server
 option transport-type tcp
 option auth.addr.brick.allow 192.168.99.*
 option transport.socket.listen-port 6999
 subvolumes brick

Client volume

volume brick-0-0
 type protocol/client
 option transport-type tcp
 option remote-host c8-0-0
 option remote-port 6999
 option remote-subvolume brick

volume brick-0-1 ...

volume rep-0
type cluster/replicate
subvolumes brick-0-0 brick-0-1 ...

volume union
type cluster/distribute
subvolumes rep-0 rep-1 rep-2 rep-3 rep-4 rep-5 rep-6 rep-7

volume client
 type performance/write-behind
 option cache-size 32MB
 option flush-behind on
 subvolumes union

For those who are interested enough to see the real configuration files, I
have all the configuration files and server/client logs uploaded to

More information about the Gluster-users mailing list