[Gluster-users] Reading from glusterfs stripe very slow

[P.Gotwalt]

Hi glusterfs users,

I am still testing stripe performance .... 

In a previous email I concluded that glusterfs-stripe works favorable for big files, because a lot of small files can introduce a lot of overhead. To test streaming performance I did a very simple write/read test.  My configuration is as follows:
- a big client machine with a 10Gb nic, 16 GB memory
- 4 glusterfs servers with 1 Gb nic's, exporting each a volume, 4 GB memory
- all connected to the same switch

Config of glusterfs 3.1.1 with a 4 node striped volume:
# gluster volume info
 
Volume Name: testvol
Type: Stripe
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: node20.storage.xx.nl:/data1
Brick2: node30.storage.xx.nl:/data1
Brick3: node40.storage.xx.nl:/data1
Brick4: node50.storage.xx.nl:/data1

I write a stream and subsequently read the same file, all with "dd". This will give me raw streaming performance. I use a file big enough file to eliminate cache effects on both the client and servers. Between the read and write I clear the OS buffer cache.

Results:
[root at x10.storage ~]# echo 3 > /proc/sys/vm/drop_caches
[root at x10.storage ~]# dd if=/dev/zero of=/gluster/file1 bs=1M count=33k
33792+0 records in
33792+0 records out
35433480192 bytes (35 GB) copied, 146.224 seconds, 242 MB/s

The write gives a nice result. I tested this also on the storage brick on a local disk and got 96MB/s. This will give a theoretical max of 384GB/s for this 4 node stripe. Ofcourse the is much more overhead, like the tcp/ip stack etc. The result of 242MB/s could be better but is ok.
The read performance of a local disk is 121 MB/s. This is impossible over the 1 Gb nics but we should get nice results too for the 4 node stripe read:

[root at x10.storage ~]# echo 3 > /proc/sys/vm/drop_caches
[root at x10.storage ~]# dd of=/dev/null if=/gluster/file1 bs=1M count=33k
33792+0 records in
33792+0 records out
35433480192 bytes (35 GB) copied, 358.679 seconds, 98.8 MB/s

And this is very disappointing! Any idea what is happening here? Because I can't believe this is normal behavior.  

Greetings

Peter Gotwalt

P.S. Didn't do any tuning on the client or server side.