[Gluster-devel] Testing --disable-direct-io-mode

[Nathan Stratton]

Configs are at:
http://www.robotics.net/glusterfs.vol
http://www.robotics.net/glusterfsd.vol

Testing with:
[root at cluster0 share]# dd if=/dev/zero of=/export/bar bs=1G count=8

Raw disk:
A 8589934592 bytes (8.6 GB) copied, 105.002 seconds, 81.8 MB/s
B 8589934592 bytes (8.6 GB) copied, 21.0523 seconds, 408 MB/s

Gluster:
A 8589934592 bytes (8.6 GB) copied, 116.116 seconds, 74.0 MB/s
B 8589934592 bytes (8.6 GB) copied, 47.4356 seconds, 181 MB/s

Gluster --disable-direct-io-mode
A 8589934592 bytes (8.6 GB) copied, 452.66 seconds, 19.0 MB/s
B 8589934592 bytes (8.6 GB) copied, 336.514 seconds, 25.5 MB/

A and B are both running distribute over replicate over ib-verbs, the only 
difference is that A just has one 750 G disk in each box and B has 16 750G 
disks in RAID6 on a 3ware 3650SE card in each.

I was happy with the just under 10% performance hit Gluster had over 
direct disk on cluster A, however but a little taken by the over 55% hit 
on cluster B. Since both clusters were connected over 10 Gbps Infiniband 
using ib-verbs I expected to get about 360 MB/s on cluster B. I did better 
then Gig E, but not much.

I also found it interesting how similar A and B were when using 
--disable-direct-io-mode. Clearly, fast local disk play little in this 
overhead.


><>
Nathan Stratton                                CTO, BlinkMind, Inc.
nathan at robotics.net                         nathan at blinkmind.com
http://www.robotics.net                        http://www.blinkmind.com