[Gluster-users] gluster rdma

[Derek Yarnell]

On 1/16/12 11:02 PM, Amar Tumballi wrote:
> 
>> [2012-01-16 19:17:41.946893] E [rdma.c:198:rdma_new_post]
>> 0-rpc-transport/rdma: memory registration failed
>> [2012-01-16 19:17:41.946932] E [rdma.c:2341:__rdma_create_posts]
>> 0-rpc-transport/rdma: rdma.gtest-server: post creation failed
>> [2012-01-16 19:17:41.948827] E [rdma.c:3861:rdma_get_device]
>> 0-rpc-transport/rdma: rdma.gtest-server: could not allocate posts
> 
> What is the memory available on the machine?
> 
> 

I have confirmed this on two different sets of hosts (Sun X4170 and a
Dell Precision WorkStation R5400).  Both sets of hosts are on the same
IB subnet with Qlogic hardware. This is all running on RHEL5.

Sun X4170
# free -m
             total       used       free     shared    buffers     cached
Mem:         24083      19643       4440          0        407      18524
-/+ buffers/cache:        711      23372
Swap:         2047          0       2047

Dell Precision WorkStation R5400
# free -m
             total       used       free     shared    buffers     cached
Mem:         16041       4512      11529          0        623       3409
-/+ buffers/cache:        479      15562
Swap:         2047          0       2047

# ib_write_bw -a 192.168.1.101
------------------------------------------------------------------
                    RDMA_Write BW Test
Number of qp's running 1
Connection type : RC
Each Qp will post up to 100 messages each time
Inline data is used up to 0 bytes message
  local address:  LID 0x09, QPN 0x000d, PSN 0x9362ce RKey 0x40a80b00
VAddr 0x002b07f6b1a000
  remote address: LID 0x20, QPN 0x000c, PSN 0x6ef146, RKey 0x40900a00
VAddr 0x002aeb16efe000
Mtu : 2048
------------------------------------------------------------------
 #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]
      2        5000               0.37                  0.37
      4        5000               0.74                  0.74
      8        5000               1.48                  1.48
     16        5000               2.98                  2.97
     32        5000               5.91                  5.91
     64        5000              11.75                 11.74
    128        5000              23.47                 23.47
    256        5000              45.79                 45.79
    512        5000              90.65                 90.63
   1024        5000             175.00                174.96
   2048        5000             322.49                320.08
   4096        5000             419.93                419.46
   8192        5000             478.97                478.96
  16384        5000             511.44                505.41
  32768        5000             516.93                516.67
  65536        5000             517.94                517.50
 131072        5000             519.76                517.61
 262144        5000             520.55                517.76
 524288        5000             518.28                517.91
1048576        5000             521.42                518.09
2097152        5000             607.73                527.41
4194304        5000             623.20                606.25
8388608        5000             617.73                607.03
------------------------------------------------------------------

-- 
---
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies