[Gluster-users] Fedora 11 - 2.6.31 Kernel - Fuse 2.8.0 - Infiniband

[Anand Avati]

> > What does the server log have to say? Can you also check if port-1 is
> > the active port in ibv_devinfo? Looks like ib-verbs messaging is not
> > happening. Does ibv_srq_pingpong give sane results?
> >
>
>  [    3.890311] ib_mthca: Mellanox InfiniBand HCA driver v1.0 (April 4,
> 2008)
>  [    3.890315] ib_mthca: Initializing 0000:08:00.0
>  [    3.890354] ib_mthca 0000:08:00.0: PCI INT A -> GSI 28 (level, low) ->
> IRQ 28
>  [    7.899804] ADDRCONF(NETDEV_UP): ib0: link is not ready
>  [    7.902722] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
>  [   17.935013] ib0: no IPv6 routers present
>
>
>  [root at xen1 ~]# ibv_devinfo
>  hca_id: mthca0
>         fw_ver:                         3.5.0
>         node_guid:                      0005:ad00:0003:27e8
>         sys_image_guid:                 0005:ad00:0100:d050
>         vendor_id:                      0x02c9
>         vendor_part_id:                 23108
>         hw_ver:                         0xA1
>         board_id:                       MT_0270110001
>         phys_port_cnt:                  2
>                 port:   1
>                         state:                  active (4)
>                         max_mtu:                2048 (4)
>                         active_mtu:             2048 (4)
>                         sm_lid:                 2
>                         port_lid:               11
>                         port_lmc:               0x00
>
>                 port:   2
>                         state:                  down (1)
>                         max_mtu:                2048 (4)
>                         active_mtu:             512 (2)
>                         sm_lid:                 0
>                         port_lid:               0
>                         port_lmc:               0x00
>
>
>
>  [root at xen1 ~]# ibv_srq_pingpong 10.13.0.220
>   local address:  LID 0x000b, QPN 0x300406, PSN 0x1ace41
>   local address:  LID 0x000b, QPN 0x300407, PSN 0x6ba197
>   local address:  LID 0x000b, QPN 0x300408, PSN 0xa6f895
>   local address:  LID 0x000b, QPN 0x300409, PSN 0xf054c0
>   local address:  LID 0x000b, QPN 0x30040a, PSN 0xea4bd3
>   local address:  LID 0x000b, QPN 0x30040b, PSN 0xfe3039
>   local address:  LID 0x000b, QPN 0x30040c, PSN 0x037fa4
>   local address:  LID 0x000b, QPN 0x30040d, PSN 0x1feccd
>   local address:  LID 0x000b, QPN 0x30040e, PSN 0x22daed
>   local address:  LID 0x000b, QPN 0x30040f, PSN 0xcaa26b
>   local address:  LID 0x000b, QPN 0x300410, PSN 0xe87f33
>   local address:  LID 0x000b, QPN 0x300411, PSN 0x84bb4a
>   local address:  LID 0x000b, QPN 0x300412, PSN 0x09286e
>   local address:  LID 0x000b, QPN 0x300413, PSN 0xecf483
>   local address:  LID 0x000b, QPN 0x300414, PSN 0xd55285
>   local address:  LID 0x000b, QPN 0x300415, PSN 0xdd7065
>   remote address: LID 0x0004, QPN 0x460406, PSN 0x198bcb
>   remote address: LID 0x0004, QPN 0x460407, PSN 0x645159
>   remote address: LID 0x0004, QPN 0x460408, PSN 0x4a1a2f
>   remote address: LID 0x0004, QPN 0x460409, PSN 0x8dff52
>   remote address: LID 0x0004, QPN 0x46040a, PSN 0xe317fd
>   remote address: LID 0x0004, QPN 0x46040b, PSN 0x12da1b
>   remote address: LID 0x0004, QPN 0x460418, PSN 0xc8e0de
>   remote address: LID 0x0004, QPN 0x460419, PSN 0xfc6e7f
>   remote address: LID 0x0004, QPN 0x46041a, PSN 0xa3ffb7
>   remote address: LID 0x0004, QPN 0x46041b, PSN 0x0cc86d
>   remote address: LID 0x0004, QPN 0x46041c, PSN 0x107a0d
>   remote address: LID 0x0004, QPN 0x46041d, PSN 0xe2661c
>   remote address: LID 0x0004, QPN 0x46041e, PSN 0xfb8fd8
>   remote address: LID 0x0004, QPN 0x46041f, PSN 0xc438a5
>   remote address: LID 0x0004, QPN 0x460420, PSN 0x0be0ff
>   remote address: LID 0x0004, QPN 0x460421, PSN 0x91b657
>
>  This one ends odd, I think it should tell me more info, but it just sits
> there.

Just starting ibv_srq_pingpong makes it the "server". You should run
"ibv_srq_pingpong <HOST>" from a second server and then the two will
ping-pong each other. Can you please post that output as well?

Avati