[Gluster-devel] Infiniband throughput

[Nathan Allen Stratton]

Anyone know idea why the wiki does not have the config files used for
client and server when benchmarks are run?

I am only getting 170 MB/s over 2 CPU quad 2.33 GHz Zeon boxes with no
load. My local storage in each box is RAID 6 over 8 disks on 3ware 9650SE
cards (PCI express). Connectivity is single port 4x Mellanox MT23108 PCIX
cards. If I write to the RAID card I get 239 MB/s.

My Configs:
http://share.robotics.net/server_vs0.vol
http://share.robotics.net/server_vs1.vol
http://share.robotics.net/server_vs2.vol
http://share.robotics.net/client.vol (same on every box)

Write to gluster share:
root at vs1.voilaip.net# time dd if=/dev/zero of=./8gbfile bs=512k
count=16384
16384+0 records in
16384+0 records out
8589934592 bytes (8.6 GB) copied, 50.5589 seconds, 170 MB/s

real    0m50.837s
user    0m0.012s
sys     0m22.341s

Write to RAID directory:
root at vs1.voilaip.net# time dd if=/dev/zero of=./8gbfile bs=512k
count=16384
16384+0 records in
16384+0 records out
8589934592 bytes (8.6 GB) copied, 35.8675 seconds, 239 MB/s

real    0m35.893s
user    0m0.024s
sys     0m20.789s

At first I thought it may be because my Infiniband cards are PCIX vs PCIe,
but when I test that I get 477 MB/s:

root at vs1.voilaip.net# ib_rdma_bw 192.168.0.12
12415: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 |
duplex=0 | cma=0 |
12415: Local address:  LID 0x03, QPN 0x10408, PSN 0x689d5f RKey 0x5f4c00bd
VAddr 0x002aaaab2f5000
12415: Remote address: LID 0x04, QPN 0x5f040a, PSN 0xce42f6, RKey
0x14fa00c0 VAddr 0x002aaaab2f5000


12415: Bandwidth peak (#0 to #954): 477.292 MB/sec
12415: Bandwidth average: 477.2 MB/sec
12415: Service Demand peak (#0 to #954): 4774 cycles/KB
12415: Service Demand Avg  : 4775 cycles/KB


-Nathan