[Gluster-users] Performance expectations

[Nathan Stratton]

On Mon, 23 Mar 2009, Gennadiy Nerubayev wrote:

> Hi all,
>
> I'm using 2.0.0rc4 on a pair of infiniband connected servers. Both servers
> are running 2.6.28.7, and are using ext4 on a local raid set as the storage
> for Gluster. The speeds that I get writing to the aforementioned ext4 mount,
> depending on benchmark and method, vary around 400MB/s (with dd) to 550MB/s+
> (with iozone, bonnie, and dd directio). Next, I used the Gluster server and
> client on both servers, with ib-verbs and replicate (see attached
> configuration files). The best speeds that I get when writing to the Gluster
> mount with this setup is about 150MB/s, going up to about 200MB/s when doing
> simultaneous writes from both nodes. I tried a couple of configuration
> tweaks such as writeback, but they don't seem to be having much effect.

That is about what I am getting:

Raw disk:
8589934592 bytes (8.6 GB) copied, 21.0523 seconds, 408 MB/s

Gluster:
8589934592 bytes (8.6 GB) copied, 47.4356 seconds, 181 MB/s

Gluster --disable-direct-io-mode
8589934592 bytes (8.6 GB) copied, 336.514 seconds, 25.5 MB/

The part that is most concerning to me is the --disable-direct-io-mode 
stuff since I am trying to use this with xen tap:aio.

> Is there something I can do to bring the performance closer to the native
> disk speed, or are there some inherent limitations somewhere that I should
> be aware of?

You can try some caching, but overall it did not help me out a lot.


><>
Nathan Stratton                                CTO, BlinkMind, Inc.
nathan at robotics.net                         nathan at blinkmind.com
http://www.robotics.net                        http://www.blinkmind.com