[Gluster-users] Performance

[Todd Daugherty]

I am not sure what you are saying? The default record size of
ib_write_bw is 64K. But here is 128K.

Do you think that the QDR Infiniband is not doing 3.3 gigabytes per
seconds? And that is why I am only getting 500 megabytes per seconds
via GlusterFS? You think that it is something wrong with the QDR
setup? What are the actual commands that you want me run on the
machines? I will run them.

Do you have a single stream GlusterFS that can go OVER 3.3 gigabytes
per second? What setup is it?

Todd

ib_write_bw -a 192.168.0.112 -s 131072
------------------------------------------------------------------
                    RDMA_Write BW Test
Number of qp's running 1
Connection type : RC
Each Qp will post up to 100 messages each time
Inline data is used up to 0 bytes message
  local address:  LID 0x0d, QPN 0x28004f, PSN 0xc85ff4 RKey 0x10045c00
VAddr 0x007f2331df1000
  remote address: LID 0x09, QPN 0x28004f, PSN 0x1f17ac, RKey
0x10045c00 VAddr 0x007f118da91000
Mtu : 2048
------------------------------------------------------------------
 #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]
      2        5000               7.62                  7.56
      4        5000              15.16                 15.14
      8        5000              30.42                 30.40
     16        5000              60.74                 60.71
     32        5000             121.48                121.40
     64        5000             241.52                241.36
    128        5000             419.01                416.47
    256        5000             791.18                790.65
    512        5000            1435.81                834.48
   1024        5000            2733.02                976.09
   2048        5000            3087.80               3086.06
   4096        5000            3164.72               3164.44
   8192        5000            3209.57               3209.36
  16384        5000            3233.73               3233.63
  32768        5000            3242.53               3242.52
  65536        5000            3248.34               3248.28
 131072        5000            3251.16               3251.13
 262144        5000            3252.69               3252.67
 524288        5000            3253.54               3253.49
1048576        5000            3253.83               3253.83
2097152        5000            3254.13               3254.12
4194304        5000            3253.54               3253.54
8388608        5000            3253.43               3253.42
------------------------------------------------------------------


did you mean this?

ib_write_bw -a 192.168.0.112 -s 134217728
------------------------------------------------------------------
                    RDMA_Write BW Test
Number of qp's running 1
Connection type : RC
Each Qp will post up to 100 messages each time
Inline data is used up to 0 bytes message
  local address:  LID 0x0d, QPN 0x2c004f, PSN 0x6b31b4 RKey 0x18045c00
VAddr 0x007f49ab29f000
  remote address: LID 0x09, QPN 0x2c004f, PSN 0xaf834b, RKey
0x18045c00 VAddr 0x007f025c876000
Mtu : 2048
------------------------------------------------------------------
 #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]
      2        5000               7.97                  7.41
      4        5000              16.00                 15.60
      8        5000              31.79                 30.78
     16        5000              63.19                 59.77
     32        5000             127.57                123.68
     64        5000             253.17                230.46
    128        5000             434.67                425.69
    256        5000             811.90                760.27
    512        5000            1359.38                821.78
   1024        5000            2710.26                917.13
   2048        5000            3085.98               3084.89
   4096        5000            3164.72               3164.34
   8192        5000            3209.07               3208.83
  16384        5000            3234.23               3234.02
  32768        5000            3242.91               3239.98
  65536        5000            3248.53               3248.51
 131072        5000            3251.13               3251.11
 262144        5000            3252.69               3252.67
 524288        5000            3253.52               3253.52
1048576        5000            3253.87               3253.85
2097152        5000            3254.15               3254.15
4194304        5000            3253.56               3253.56
8388608        5000            3253.44               3253.44
------------------------------------------------------------------