[Gluster-users] Gluster 3.3.0 on CentOS 6 - GigabitEthernet vs InfiniBand

[Corey Kovacs]

One problem right away is that 3.3 doesn't support rdma. I know it still
builds the rdma packages but rdma isn't a supported connection method for
3.3. It was set aside so the 33. release could make it on time. As I
understand it, 3.1 was/is suppposed to be the first 3.x version to fully
support it. I'd upgrade and test again. That or downgrade to 3.2.7 which
does in fact support it right now.

I'm not even sure how you got things mounted using the "rdma" semantics
with 3.3.

My experiences so far were sort of disappointing until I found out a few
key items about GlusterFS which I'd taken for granted.

1. Stripes are not what you might think. The I/O for a stripe does _not_
fan out as in a raid card. It's an unfortunate use of the term only
describing and allowing you to store files larger than the max size of a
single brick.
2. I/O is done in sync mode so cache coherency isn't an issue and to ensure
the integrity of the data written.
3. The performance of a distributed volume far exceeds that of a stripe for
my use. Again, depends on the size of the bricks.

These are things which effect my experience and now that I at least _think_
I understand them my results make much more sense to me. Generally my
throughput maxes out around 800-900MB/s which is the limit of my disk
storage right now.

As a test I created as large of a volume as I could using ramdisks. I did
this to see just how much of a limiter my disks actually are, and I was
very surprised to see the speed NOT increase across the file system using
an rdma target on version 3.2.6. The local I/O reached 1.9GB/s. So, in
addition to my spindle based limit, I do not believe I am using any more
than a single lane of my IB cards (which are FDR ( 56Gb) Mellanox).

Using the ib test utilities I can indeed max the connection at 6GB/s, which
is really amazing to see but so far I just can't seem to get GlusterFS to
make use of the total B/W available.

Also, you might try putting your cads into "Connected Mode" and increasing
your MTU to 64k.


Corey
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121018/08e94510/attachment.html>