[Gluster-users] Slow reading speed over RDMA

Daniel Pereira d.pereira at skillupjapan.co.jp
Thu May 12 10:35:06 UTC 2011

 Hi everyone,

 I hope you can help me with some performance troubles I've been having.
 I'm doing some tests with gluster 3.2.0, and I can't understand some of the
behavior I'm getting with gluster.

 The test is using a volume with 12 stripped bricks (each brick is an HD),
with no replication, via RDMA. I'm doing random readings of 4GByte files
with the FIO tool.
 Gluster reading speed is of 150-160MByte/s and 110-120MByte/s per client if
I connect with two clients.

 However, on the same machine, I have a similar lustre setup with a constant
400MByte/s throughput, for the same tests, with the same amount of disks
going through the same RAID controller. Both tests were ran at different

 As comparison, for the sequential reading tests, both gluster and lustre
give me, respectively, results of 600 to 700MByte/s.

 The gluster configuration files have no extra modifications, the disks are
formatted with ext3 and created via:
gluster volume create test stripe 16 transport rdma ...

 Using DD to write/read from each of the disks gives me about 100MByte/s.

 Would there be anything obvious I'm missing? What would be causing this low
reading speed? I tried playing with the performance parameters, setting up
more or less cache and io-threads, but the results did not improve.
Moreover, I have tested all the suggested optimization hacks/parameters with
little change in the results.

 A little more insight on the setup: we have 24 HD machines, with dual Xeon
CPUs, 16GByte of RAM, working over an Areca RAID card, using JBOD, all boxes
connected via 10Gbit/s Infiniband. The CPUs are far from being maxed out
(top reports around 50% idle), as is the network.
 I'm using CentOS 5.5 on all the machines. Let me know if you need more
information on what's going on (configuration files, setup, etc, etc).

 Thanks in advance,

 SkillUpJapan Corporation
 Research and Development Office
  Senior Researcher Engineer
 Daniel Pereira d.pereira at skillupjapan.co.jp
 Tokyo, Shinjuku, Takadanobaba 1-24-16
   Uchida Building 1st Floor
 TEL:03-5287-4087 FAX:03-5287-4135

More information about the Gluster-users mailing list