[Gluster-users] Fedora 11 - 2.6.31 Kernel - Fuse 2.8.0 - Infiniband
Nathan Stratton
nathan at robotics.net
Tue Sep 22 18:23:21 UTC 2009
On Tue, 22 Sep 2009, Anand Avati wrote:
>>> What does the server log have to say? Can you also check if port-1 is
>>> the active port in ibv_devinfo? Looks like ib-verbs messaging is not
>>> happening. Does ibv_srq_pingpong give sane results?
>>>
>>
>> [ 3.890311] ib_mthca: Mellanox InfiniBand HCA driver v1.0 (April 4,
>> 2008)
>> [ 3.890315] ib_mthca: Initializing 0000:08:00.0
>> [ 3.890354] ib_mthca 0000:08:00.0: PCI INT A -> GSI 28 (level, low) ->
>> IRQ 28
>> [ 7.899804] ADDRCONF(NETDEV_UP): ib0: link is not ready
>> [ 7.902722] ADDRCONF(NETDEV_CHANGE): ib0: link becomes ready
>> [ 17.935013] ib0: no IPv6 routers present
>>
>>
>> [root at xen1 ~]# ibv_devinfo
>> hca_id: mthca0
>> fw_ver: 3.5.0
>> node_guid: 0005:ad00:0003:27e8
>> sys_image_guid: 0005:ad00:0100:d050
>> vendor_id: 0x02c9
>> vendor_part_id: 23108
>> hw_ver: 0xA1
>> board_id: MT_0270110001
>> phys_port_cnt: 2
>> port: 1
>> state: active (4)
>> max_mtu: 2048 (4)
>> active_mtu: 2048 (4)
>> sm_lid: 2
>> port_lid: 11
>> port_lmc: 0x00
>>
>> port: 2
>> state: down (1)
>> max_mtu: 2048 (4)
>> active_mtu: 512 (2)
>> sm_lid: 0
>> port_lid: 0
>> port_lmc: 0x00
>>
>>
>>
>> [root at xen1 ~]# ibv_srq_pingpong 10.13.0.220
>> local address: LID 0x000b, QPN 0x300406, PSN 0x1ace41
>> local address: LID 0x000b, QPN 0x300407, PSN 0x6ba197
>> local address: LID 0x000b, QPN 0x300408, PSN 0xa6f895
>> local address: LID 0x000b, QPN 0x300409, PSN 0xf054c0
>> local address: LID 0x000b, QPN 0x30040a, PSN 0xea4bd3
>> local address: LID 0x000b, QPN 0x30040b, PSN 0xfe3039
>> local address: LID 0x000b, QPN 0x30040c, PSN 0x037fa4
>> local address: LID 0x000b, QPN 0x30040d, PSN 0x1feccd
>> local address: LID 0x000b, QPN 0x30040e, PSN 0x22daed
>> local address: LID 0x000b, QPN 0x30040f, PSN 0xcaa26b
>> local address: LID 0x000b, QPN 0x300410, PSN 0xe87f33
>> local address: LID 0x000b, QPN 0x300411, PSN 0x84bb4a
>> local address: LID 0x000b, QPN 0x300412, PSN 0x09286e
>> local address: LID 0x000b, QPN 0x300413, PSN 0xecf483
>> local address: LID 0x000b, QPN 0x300414, PSN 0xd55285
>> local address: LID 0x000b, QPN 0x300415, PSN 0xdd7065
>> remote address: LID 0x0004, QPN 0x460406, PSN 0x198bcb
>> remote address: LID 0x0004, QPN 0x460407, PSN 0x645159
>> remote address: LID 0x0004, QPN 0x460408, PSN 0x4a1a2f
>> remote address: LID 0x0004, QPN 0x460409, PSN 0x8dff52
>> remote address: LID 0x0004, QPN 0x46040a, PSN 0xe317fd
>> remote address: LID 0x0004, QPN 0x46040b, PSN 0x12da1b
>> remote address: LID 0x0004, QPN 0x460418, PSN 0xc8e0de
>> remote address: LID 0x0004, QPN 0x460419, PSN 0xfc6e7f
>> remote address: LID 0x0004, QPN 0x46041a, PSN 0xa3ffb7
>> remote address: LID 0x0004, QPN 0x46041b, PSN 0x0cc86d
>> remote address: LID 0x0004, QPN 0x46041c, PSN 0x107a0d
>> remote address: LID 0x0004, QPN 0x46041d, PSN 0xe2661c
>> remote address: LID 0x0004, QPN 0x46041e, PSN 0xfb8fd8
>> remote address: LID 0x0004, QPN 0x46041f, PSN 0xc438a5
>> remote address: LID 0x0004, QPN 0x460420, PSN 0x0be0ff
>> remote address: LID 0x0004, QPN 0x460421, PSN 0x91b657
>>
>> This one ends odd, I think it should tell me more info, but it just sits
>> there.
>
> Just starting ibv_srq_pingpong makes it the "server". You should run
> "ibv_srq_pingpong <HOST>" from a second server and then the two will
> ping-pong each other. Can you please post that output as well?
The above was from the client using the IP address of the server. The
server showed:
[root at xen0 ~]# ibv_srq_pingpong
local address: LID 0x0004, QPN 0x460406, PSN 0x198bcb
local address: LID 0x0004, QPN 0x460407, PSN 0x645159
local address: LID 0x0004, QPN 0x460408, PSN 0x4a1a2f
local address: LID 0x0004, QPN 0x460409, PSN 0x8dff52
local address: LID 0x0004, QPN 0x46040a, PSN 0xe317fd
local address: LID 0x0004, QPN 0x46040b, PSN 0x12da1b
local address: LID 0x0004, QPN 0x460418, PSN 0xc8e0de
local address: LID 0x0004, QPN 0x460419, PSN 0xfc6e7f
local address: LID 0x0004, QPN 0x46041a, PSN 0xa3ffb7
local address: LID 0x0004, QPN 0x46041b, PSN 0x0cc86d
local address: LID 0x0004, QPN 0x46041c, PSN 0x107a0d
local address: LID 0x0004, QPN 0x46041d, PSN 0xe2661c
local address: LID 0x0004, QPN 0x46041e, PSN 0xfb8fd8
local address: LID 0x0004, QPN 0x46041f, PSN 0xc438a5
local address: LID 0x0004, QPN 0x460420, PSN 0x0be0ff
local address: LID 0x0004, QPN 0x460421, PSN 0x91b657
remote address: LID 0x000b, QPN 0x300406, PSN 0x1ace41
remote address: LID 0x000b, QPN 0x300407, PSN 0x6ba197
remote address: LID 0x000b, QPN 0x300408, PSN 0xa6f895
remote address: LID 0x000b, QPN 0x300409, PSN 0xf054c0
remote address: LID 0x000b, QPN 0x30040a, PSN 0xea4bd3
remote address: LID 0x000b, QPN 0x30040b, PSN 0xfe3039
remote address: LID 0x000b, QPN 0x30040c, PSN 0x037fa4
remote address: LID 0x000b, QPN 0x30040d, PSN 0x1feccd
remote address: LID 0x000b, QPN 0x30040e, PSN 0x22daed
remote address: LID 0x000b, QPN 0x30040f, PSN 0xcaa26b
remote address: LID 0x000b, QPN 0x300410, PSN 0xe87f33
remote address: LID 0x000b, QPN 0x300411, PSN 0x84bb4a
remote address: LID 0x000b, QPN 0x300412, PSN 0x09286e
remote address: LID 0x000b, QPN 0x300413, PSN 0xecf483
remote address: LID 0x000b, QPN 0x300414, PSN 0xd55285
remote address: LID 0x000b, QPN 0x300415, PSN 0xdd7065
-Nathan
More information about the Gluster-users
mailing list