[Gluster-devel] Glusterfs setup+benchmark (slow?)

Anand Avati avati at zresearch.com
Fri May 30 05:01:39 UTC 2008


>
> Trying to get som idea of the performance of a glusterfs setup.
> I'm not very happy with the numbers I have got from my tests.
> I would like to check with you folks and see if my setup/numbers
> looks correct...
>
> The setup consists of 3 glusterfs servers using 1G RAM
> (boot param mem=1024M)
>
> Hw setup (3 server 1 client):
> ------------------------------
>   - 2 x Intel(R) Xeon(R) CPU E5420  @ 2.50GHz (8 cores)
>   - Intel Corporation 80003ES2LAN Gigabit Ethernet Controller
>   - ATA-7: ST380815AS, 3.AAD, max UDMA/133
>
>
> Versions used:
> ---------------
>   glusterfs: glusterfs-1.3.8pre5
>   fuse: fuse-2.7.2glfs9
>   CentOS: 5
>
>
> Local disk timings (hdparm -Tt):
> --------------------------------
>   Timing cached reads:   24636 MB in  1.99 seconds = 12353.07 MB/sec
>
>
> Network Throughput:
> -------------------
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    MBytes/sec
> 87380  16384  16384    180.03    112.23
>
>
> Server config:
> ------------------------------------------------------
> volume brick
>   type storage/posix
>   option directory /data/export
> end-volume
>
> volume brick-ns
>   type storage/posix
>   option directory /data/export-ns
> end-volume
>
> volume server
>   type protocol/server
>   option transport-type tcp/server
>   option auth.ip.brick.allow *
>   option auth.ip.brick-ns.allow *
>   subvolumes brick brick-ns
> end-volume
> ------------------------------------------------------
>
>
> Client config (unify):
> ------------------------------------------------------
> volume remote1
>   type protocol/client
>   option transport-type tcp/client
>   option remote-host 10.10.10.164
>   option remote-subvolume brick
> end-volume
>
> volume remote2
>   type protocol/client
>   option transport-type tcp/client
>   option remote-host 10.10.10.197
>   option remote-subvolume brick
> end-volume
>
> volume remote3
>   type protocol/client
>   option transport-type tcp/client
>   option remote-host 10.10.10.46
>   option remote-subvolume brick
> end-volume
>
> volume remote-ns
>   type protocol/client
>   option transport-type tcp/client
>   option remote-host 10.10.10.164
>   option remote-subvolume brick-ns
> end-volume
>
> volume unify0
>   type cluster/unify
>   option scheduler rr
>   option namespace remote-ns
>   subvolumes remote1 remote2 remote3
> end-volume
> ------------------------------------------------------
>
>
> Client config (stripe):
> ------------------------------------------------------
> volume remote1
>   type protocol/client
>   option transport-type tcp/client
>   option remote-host 10.10.10.164
>   option remote-subvolume brick
> end-volume
>
> volume remote2
>   type protocol/client
>   option transport-type tcp/client
>   option remote-host 10.10.10.197
>   option remote-subvolume brick
> end-volume
>
> volume remote3
>   type protocol/client
>   option transport-type tcp/client
>   option remote-host 10.10.10.46
>   option remote-subvolume brick
> end-volume
>
> volume stripe
>    type cluster/stripe
>    option block-size *:1MB
>    subvolumes remote1 remote2 remote3
>   end-volume
> ------------------------------------------------------
>
> These simple tests was made with dd and in the case
> with 3 concurrent processes first 3 concurrent write
> and then 3 concurrent write.
>
>
> Test #1 (1 client 3 servers 1 write process) Stripe
> ------------------------------------------------------
>
> Write test:
>   * 10485760000 bytes (10 GB) copied, 128.899 seconds, 81.3 MB/s
>   * 10485760000 bytes (10 GB) copied, 128.435 seconds, 81.6 MB/s
>   * 10485760000 bytes (10 GB) copied, 134.973 seconds, 77.7 MB/s
>
> Read test:
>   * 10485760000 bytes (10 GB) copied, 530.27 seconds, 19.8 MB/s
>   * 10485760000 bytes (10 GB) copied, 455.247 seconds, 23.0 MB/s
>   * 10485760000 bytes (10 GB) copied, 449.548 seconds, 23.3 MB/s
>
>
>
> Test #2 (1 client 3 servers 3 concurrent processes) Stripe
> ------------------------------------------------------
> Process 1:
> ----------
> Write:
>   * 5242880000 bytes (5.2 GB) copied, 584.393 seconds, 9.0 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 583.894 seconds, 9.0 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 588.164 seconds, 8.9 MB/s
>
> Read:
>   * 5242880000 bytes (5.2 GB) copied, 386.279 seconds, 13.6 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 385.255 seconds, 13.6 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 386.346 seconds, 13.6 MB/s
>
> Process 2:
> ----------
> Write
>   * 5242880000 bytes (5.2 GB) copied, 587.611 seconds, 8.9 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 589.912 seconds, 8.9 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 605.053 seconds, 8.7 MB/s :
>
> Read:
>   * 5242880000 bytes (5.2 GB) copied, 411.217 seconds, 12.7 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 386.303 seconds, 13.6 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 386.303 seconds, 13.6 MB/s
>
> Process 3:
> ----------
> Write:
>   * 5242880000 bytes (5.2 GB) copied, 587.612 seconds, 8.9 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 589.902 seconds, 8.9 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 605.063 seconds, 8.7 MB/s
>
> Read:
>   * 5242880000 bytes (5.2 GB) copied, 411.217 seconds, 12.7 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 386.303 seconds, 13.6 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 386.303 seconds, 13.6 MB/s
>
>
> Test #3 (1 client 3 server 2 write then 2 read processes) Unify
> ----------------------------------------------------------------
> (only two tests runs ran out of time with hw)
> Process 1:
> ----------
> Write:
>   * 5242880000 bytes (5.2 GB) copied, 858.661 seconds, 6.1 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 218.973 seconds, 23.9 MB/s
>
> Read:
>   * 5242880000 bytes (5.2 GB) copied, 447.404 seconds, 11.7 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 432.071 seconds, 12.1 MB/s
>
> Process 2:
> ----------
> Write:
>   * 5242880000 bytes (5.2 GB) copied, 214.628 seconds, 24.4 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 483.334 seconds, 10.8 MB/s
>
> Read:
>   * 5242880000 bytes (5.2 GB) copied, 320.471 seconds, 16.4 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 662.589 seconds, 7.9 MB/s
>
> Process 3:
> ----------
> Write:
>   * 5242880000 bytes (5.2 GB) copied, 214.799 seconds, 24.4 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 809.602 seconds, 6.5 MB/s
>
> Read:
>   * 5242880000 bytes (5.2 GB) copied, 660.162 seconds, 7.9 MB/s
>   * 5242880000 bytes (5.2 GB) copied, 458.078 seconds, 11.4 MB/s
>
> # End of tests
>
>
> What do you think? Do my numbers and setup look sane?



Some questions -

- what was the block size used during the dd tests?

- You might want to load io-threads and read-ahead on the server and
write-behind on the client spec files.

- There is a new non-blocking transport introduced in
glusterfs--mainline--3.0 which improves performance in multi client/server
mode. You might be interested to try it out.

avati



More information about the Gluster-devel mailing list