[Gluster-users] RDMA Problems with GlusterFS 3.1.1
Jeremy Stout
stout.jeremy at gmail.com
Fri Dec 3 12:20:04 UTC 2010
I patched the source code and rebuilt GlusterFS. Here are the full logs:
Server:
[2010-12-03 07:08:55.945804] I [glusterd.c:275:init] management: Using
/etc/glusterd as working directory
[2010-12-03 07:08:55.947692] E [rdma.c:2047:rdma_create_cq]
rpc-transport/rdma: max_mr_size = 18446744073709551615, max_cq =
65408, max_cqe = 131071, max_mr = 131056
[2010-12-03 07:08:55.953226] E [rdma.c:2079:rdma_create_cq]
rpc-transport/rdma: rdma.management: creation of send_cq failed
[2010-12-03 07:08:55.953509] E [rdma.c:3785:rdma_get_device]
rpc-transport/rdma: rdma.management: could not create CQ
[2010-12-03 07:08:55.953582] E [rdma.c:3971:rdma_init]
rpc-transport/rdma: could not create rdma device for mthca0
[2010-12-03 07:08:55.953668] E [rdma.c:4803:init] rdma.management:
Failed to initialize IB Device
[2010-12-03 07:08:55.953691] E
[rpc-transport.c:971:rpc_transport_load] rpc-transport: 'rdma'
initialization failed
[2010-12-03 07:08:55.953780] I [glusterd.c:96:glusterd_uuid_init]
glusterd: generated UUID: 4eb47ca7-227c-49c4-97bd-25ac177b2f6a
Given volfile:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option working-directory /etc/glusterd
4: option transport-type socket,rdma
5: option transport.socket.keepalive-time 10
6: option transport.socket.keepalive-interval 2
7: end-volume
8:
+------------------------------------------------------------------------------+
[2010-12-03 07:09:10.244790] I
[glusterd-handler.c:785:glusterd_handle_create_volume] glusterd:
Received create volume req
[2010-12-03 07:09:10.247646] I [glusterd-utils.c:232:glusterd_lock]
glusterd: Cluster lock held by 4eb47ca7-227c-49c4-97bd-25ac177b2f6a
[2010-12-03 07:09:10.247678] I
[glusterd-handler.c:2835:glusterd_op_txn_begin] glusterd: Acquired
local lock
[2010-12-03 07:09:10.247708] I
[glusterd3_1-mops.c:1091:glusterd3_1_cluster_lock] glusterd: Sent lock
req to 0 peers
[2010-12-03 07:09:10.248038] I
[glusterd3_1-mops.c:1233:glusterd3_1_stage_op] glusterd: Sent op req
to 0 peers
[2010-12-03 07:09:10.251970] I
[glusterd3_1-mops.c:1323:glusterd3_1_commit_op] glusterd: Sent op req
to 0 peers
[2010-12-03 07:09:10.252020] I
[glusterd3_1-mops.c:1145:glusterd3_1_cluster_unlock] glusterd: Sent
unlock req to 0 peers
[2010-12-03 07:09:10.252036] I
[glusterd-op-sm.c:4738:glusterd_op_txn_complete] glusterd: Cleared
local lock
[2010-12-03 07:09:22.11649] I
[glusterd-handler.c:936:glusterd_handle_cli_start_volume] glusterd:
Received start vol reqfor volume testdir
[2010-12-03 07:09:22.11724] I [glusterd-utils.c:232:glusterd_lock]
glusterd: Cluster lock held by 4eb47ca7-227c-49c4-97bd-25ac177b2f6a
[2010-12-03 07:09:22.11734] I
[glusterd-handler.c:2835:glusterd_op_txn_begin] glusterd: Acquired
local lock
[2010-12-03 07:09:22.11761] I
[glusterd3_1-mops.c:1091:glusterd3_1_cluster_lock] glusterd: Sent lock
req to 0 peers
[2010-12-03 07:09:22.12120] I
[glusterd3_1-mops.c:1233:glusterd3_1_stage_op] glusterd: Sent op req
to 0 peers
[2010-12-03 07:09:22.184403] I
[glusterd-utils.c:971:glusterd_volume_start_glusterfs] : About to
start glusterfs for brick pgh-submit-1:/mnt/gluster
[2010-12-03 07:09:22.229143] I
[glusterd3_1-mops.c:1323:glusterd3_1_commit_op] glusterd: Sent op req
to 0 peers
[2010-12-03 07:09:22.229198] I
[glusterd3_1-mops.c:1145:glusterd3_1_cluster_unlock] glusterd: Sent
unlock req to 0 peers
[2010-12-03 07:09:22.229218] I
[glusterd-op-sm.c:4738:glusterd_op_txn_complete] glusterd: Cleared
local lock
[2010-12-03 07:09:22.240157] I
[glusterd-pmap.c:281:pmap_registry_remove] pmap: removing brick (null)
on port 24009
Client:
[2010-12-03 07:09:00.82784] W [io-stats.c:1644:init] testdir: dangling
volume. check volfile
[2010-12-03 07:09:00.82824] W [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-12-03 07:09:00.82836] W [dict.c:1204:data_to_str] dict: @data=(nil)
[2010-12-03 07:09:00.85980] E [rdma.c:2047:rdma_create_cq]
rpc-transport/rdma: max_mr_size = 18446744073709551615, max_cq =
65408, max_cqe = 131071, max_mr = 131056
[2010-12-03 07:09:00.92883] E [rdma.c:2079:rdma_create_cq]
rpc-transport/rdma: testdir-client-0: creation of send_cq failed
[2010-12-03 07:09:00.93156] E [rdma.c:3785:rdma_get_device]
rpc-transport/rdma: testdir-client-0: could not create CQ
[2010-12-03 07:09:00.93224] E [rdma.c:3971:rdma_init]
rpc-transport/rdma: could not create rdma device for mthca0
[2010-12-03 07:09:00.93313] E [rdma.c:4803:init] testdir-client-0:
Failed to initialize IB Device
[2010-12-03 07:09:00.93332] E [rpc-transport.c:971:rpc_transport_load]
rpc-transport: 'rdma' initialization failed
Given volfile:
+------------------------------------------------------------------------------+
1: volume testdir-client-0
2: type protocol/client
3: option remote-host submit-1
4: option remote-subvolume /mnt/gluster
5: option transport-type rdma
6: end-volume
7:
8: volume testdir-write-behind
9: type performance/write-behind
10: subvolumes testdir-client-0
11: end-volume
12:
13: volume testdir-read-ahead
14: type performance/read-ahead
15: subvolumes testdir-write-behind
16: end-volume
17:
18: volume testdir-io-cache
19: type performance/io-cache
20: subvolumes testdir-read-ahead
21: end-volume
22:
23: volume testdir-quick-read
24: type performance/quick-read
25: subvolumes testdir-io-cache
26: end-volume
27:
28: volume testdir-stat-prefetch
29: type performance/stat-prefetch
30: subvolumes testdir-quick-read
31: end-volume
32:
33: volume testdir
34: type debug/io-stats
35: subvolumes testdir-stat-prefetch
36: end-volume
+------------------------------------------------------------------------------+
On Fri, Dec 3, 2010 at 12:38 AM, Raghavendra G <raghavendra at gluster.com> wrote:
> Hi Jeremy,
>
> Can you apply the attached patch, rebuild and start glusterfs? Please make sure to send us the logs of glusterfs.
>
> regards,
> ----- Original Message -----
> From: "Jeremy Stout" <stout.jeremy at gmail.com>
> To: gluster-users at gluster.org
> Sent: Friday, December 3, 2010 6:38:00 AM
> Subject: Re: [Gluster-users] RDMA Problems with GlusterFS 3.1.1
>
> I'm currently using OFED 1.5.2.
>
> For the sake of testing, I just compiled GlusterFS 3.1.1 from source,
> without any modifications, on two systems that have a 2.6.33.7 kernel
> and OFED 1.5.2 built from source. Here are the results:
>
> Server:
> [2010-12-02 21:17:55.886563] I
> [glusterd-handler.c:936:glusterd_handle_cli_start_volume] glusterd:
> Received start vol reqfor volume testdir
> [2010-12-02 21:17:55.886597] I [glusterd-utils.c:232:glusterd_lock]
> glusterd: Cluster lock held by 7dd23af5-277e-4ea1-a495-2a9d882287ec
> [2010-12-02 21:17:55.886607] I
> [glusterd-handler.c:2835:glusterd_op_txn_begin] glusterd: Acquired
> local lock
> [2010-12-02 21:17:55.886628] I
> [glusterd3_1-mops.c:1091:glusterd3_1_cluster_lock] glusterd: Sent lock
> req to 0 peers
> [2010-12-02 21:17:55.887031] I
> [glusterd3_1-mops.c:1233:glusterd3_1_stage_op] glusterd: Sent op req
> to 0 peers
> [2010-12-02 21:17:56.60427] I
> [glusterd-utils.c:971:glusterd_volume_start_glusterfs] : About to
> start glusterfs for brick submit-1:/mnt/gluster
> [2010-12-02 21:17:56.104896] I
> [glusterd3_1-mops.c:1323:glusterd3_1_commit_op] glusterd: Sent op req
> to 0 peers
> [2010-12-02 21:17:56.104935] I
> [glusterd3_1-mops.c:1145:glusterd3_1_cluster_unlock] glusterd: Sent
> unlock req to 0 peers
> [2010-12-02 21:17:56.104953] I
> [glusterd-op-sm.c:4738:glusterd_op_txn_complete] glusterd: Cleared
> local lock
> [2010-12-02 21:17:56.114764] I
> [glusterd-pmap.c:281:pmap_registry_remove] pmap: removing brick (null)
> on port 24009
>
> Client:
> [2010-12-02 21:17:25.503395] W [io-stats.c:1644:init] testdir:
> dangling volume. check volfile
> [2010-12-02 21:17:25.503434] W [dict.c:1204:data_to_str] dict: @data=(nil)
> [2010-12-02 21:17:25.503447] W [dict.c:1204:data_to_str] dict: @data=(nil)
> [2010-12-02 21:17:25.543409] E [rdma.c:2066:rdma_create_cq]
> rpc-transport/rdma: testdir-client-0: creation of send_cq failed
> [2010-12-02 21:17:25.543660] E [rdma.c:3771:rdma_get_device]
> rpc-transport/rdma: testdir-client-0: could not create CQ
> [2010-12-02 21:17:25.543725] E [rdma.c:3957:rdma_init]
> rpc-transport/rdma: could not create rdma device for mthca0
> [2010-12-02 21:17:25.543812] E [rdma.c:4789:init] testdir-client-0:
> Failed to initialize IB Device
> [2010-12-02 21:17:25.543830] E
> [rpc-transport.c:971:rpc_transport_load] rpc-transport: 'rdma'
> initialization failed
>
> Thank you for the help so far.
>
> On Thu, Dec 2, 2010 at 8:02 PM, Craig Carl <craig at gluster.com> wrote:
>> Jeremy -
>> What version of OFED are you running? Would you mind install version 1.5.2
>> from source? We have seen this resolve several issues of this type.
>> http://www.openfabrics.org/downloads/OFED/ofed-1.5.2/
>>
>>
>> Thanks,
>>
>> Craig
>>
>> -->
>> Craig Carl
>> Senior Systems Engineer
>> Gluster
>>
>>
>> On 12/02/2010 10:05 AM, Jeremy Stout wrote:
>>>
>>> An another follow-up, I tested several compilations today with
>>> different values for send/receive count. I found the maximum value I
>>> could use for both variables was 127. With a value of 127, GlusterFS
>>> did not produce any errors. However, when I changed the value back to
>>> 128, the RDMA errors appeared again.
>>>
>>> I also tried setting soft/hard "memlock" to unlimited in the
>>> limits.conf file, but still ran into RDMA errors on the client side
>>> when the count variables were set to 128.
>>>
>>> On Thu, Dec 2, 2010 at 9:04 AM, Jeremy Stout<stout.jeremy at gmail.com>
>>> wrote:
>>>>
>>>> Thank you for the response. I've been testing GlusterFS 3.1.1 on two
>>>> different OpenSUSE 11.3 systems. Since both systems generated the same
>>>> error messages, I'll include the output for both.
>>>>
>>>> System #1:
>>>> fs-1:~ # cat /proc/meminfo
>>>> MemTotal: 16468756 kB
>>>> MemFree: 16126680 kB
>>>> Buffers: 15680 kB
>>>> Cached: 155860 kB
>>>> SwapCached: 0 kB
>>>> Active: 65228 kB
>>>> Inactive: 123100 kB
>>>> Active(anon): 18632 kB
>>>> Inactive(anon): 48 kB
>>>> Active(file): 46596 kB
>>>> Inactive(file): 123052 kB
>>>> Unevictable: 1988 kB
>>>> Mlocked: 1988 kB
>>>> SwapTotal: 0 kB
>>>> SwapFree: 0 kB
>>>> Dirty: 30072 kB
>>>> Writeback: 4 kB
>>>> AnonPages: 18780 kB
>>>> Mapped: 12136 kB
>>>> Shmem: 220 kB
>>>> Slab: 39592 kB
>>>> SReclaimable: 13108 kB
>>>> SUnreclaim: 26484 kB
>>>> KernelStack: 2360 kB
>>>> PageTables: 2036 kB
>>>> NFS_Unstable: 0 kB
>>>> Bounce: 0 kB
>>>> WritebackTmp: 0 kB
>>>> CommitLimit: 8234376 kB
>>>> Committed_AS: 107304 kB
>>>> VmallocTotal: 34359738367 kB
>>>> VmallocUsed: 314316 kB
>>>> VmallocChunk: 34349860776 kB
>>>> HardwareCorrupted: 0 kB
>>>> HugePages_Total: 0
>>>> HugePages_Free: 0
>>>> HugePages_Rsvd: 0
>>>> HugePages_Surp: 0
>>>> Hugepagesize: 2048 kB
>>>> DirectMap4k: 9856 kB
>>>> DirectMap2M: 3135488 kB
>>>> DirectMap1G: 13631488 kB
>>>>
>>>> fs-1:~ # uname -a
>>>> Linux fs-1 2.6.32.25-November2010 #2 SMP PREEMPT Mon Nov 1 15:19:55
>>>> EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
>>>>
>>>> fs-1:~ # ulimit -l
>>>> 64
>>>>
>>>> System #2:
>>>> submit-1:~ # cat /proc/meminfo
>>>> MemTotal: 16470424 kB
>>>> MemFree: 16197292 kB
>>>> Buffers: 11788 kB
>>>> Cached: 85492 kB
>>>> SwapCached: 0 kB
>>>> Active: 39120 kB
>>>> Inactive: 76548 kB
>>>> Active(anon): 18532 kB
>>>> Inactive(anon): 48 kB
>>>> Active(file): 20588 kB
>>>> Inactive(file): 76500 kB
>>>> Unevictable: 0 kB
>>>> Mlocked: 0 kB
>>>> SwapTotal: 67100656 kB
>>>> SwapFree: 67100656 kB
>>>> Dirty: 24 kB
>>>> Writeback: 0 kB
>>>> AnonPages: 18408 kB
>>>> Mapped: 11644 kB
>>>> Shmem: 184 kB
>>>> Slab: 34000 kB
>>>> SReclaimable: 8512 kB
>>>> SUnreclaim: 25488 kB
>>>> KernelStack: 2160 kB
>>>> PageTables: 1952 kB
>>>> NFS_Unstable: 0 kB
>>>> Bounce: 0 kB
>>>> WritebackTmp: 0 kB
>>>> CommitLimit: 75335868 kB
>>>> Committed_AS: 105620 kB
>>>> VmallocTotal: 34359738367 kB
>>>> VmallocUsed: 76416 kB
>>>> VmallocChunk: 34359652640 kB
>>>> HardwareCorrupted: 0 kB
>>>> HugePages_Total: 0
>>>> HugePages_Free: 0
>>>> HugePages_Rsvd: 0
>>>> HugePages_Surp: 0
>>>> Hugepagesize: 2048 kB
>>>> DirectMap4k: 7488 kB
>>>> DirectMap2M: 16769024 kB
>>>>
>>>> submit-1:~ # uname -a
>>>> Linux submit-1 2.6.33.7-November2010 #1 SMP PREEMPT Mon Nov 8 13:49:00
>>>> EST 2010 x86_64 x86_64 x86_64 GNU/Linux
>>>>
>>>> submit-1:~ # ulimit -l
>>>> 64
>>>>
>>>> I retrieved the memory information on each machine after starting the
>>>> glusterd process.
>>>>
>>>> On Thu, Dec 2, 2010 at 3:51 AM, Raghavendra G<raghavendra at gluster.com>
>>>> wrote:
>>>>>
>>>>> Hi Jeremy,
>>>>>
>>>>> can you also get the output of,
>>>>>
>>>>> #uname -a
>>>>>
>>>>> #ulimit -l
>>>>>
>>>>> regards,
>>>>> ----- Original Message -----
>>>>> From: "Raghavendra G"<raghavendra at gluster.com>
>>>>> To: "Jeremy Stout"<stout.jeremy at gmail.com>
>>>>> Cc: gluster-users at gluster.org
>>>>> Sent: Thursday, December 2, 2010 10:20:04 AM
>>>>> Subject: Re: [Gluster-users] RDMA Problems with GlusterFS 3.1.1
>>>>>
>>>>> Hi Jeremy,
>>>>>
>>>>> In order to diagnoise why completion queue creation is failing (as
>>>>> indicated by logs), we want to know what was the free memory available in
>>>>> your system when glusterfs was started.
>>>>>
>>>>> regards,
>>>>> ----- Original Message -----
>>>>> From: "Raghavendra G"<raghavendra at gluster.com>
>>>>> To: "Jeremy Stout"<stout.jeremy at gmail.com>
>>>>> Cc: gluster-users at gluster.org
>>>>> Sent: Thursday, December 2, 2010 10:11:18 AM
>>>>> Subject: Re: [Gluster-users] RDMA Problems with GlusterFS 3.1.1
>>>>>
>>>>> Hi Jeremy,
>>>>>
>>>>> Yes, there might be some performance decrease. But, it should not affect
>>>>> working of rdma.
>>>>>
>>>>> regards,
>>>>> ----- Original Message -----
>>>>> From: "Jeremy Stout"<stout.jeremy at gmail.com>
>>>>> To: gluster-users at gluster.org
>>>>> Sent: Thursday, December 2, 2010 8:30:20 AM
>>>>> Subject: Re: [Gluster-users] RDMA Problems with GlusterFS 3.1.1
>>>>>
>>>>> As an update to my situation, I think I have GlusterFS 3.1.1 working
>>>>> now. I was able to create and mount RDMA volumes without any errors.
>>>>>
>>>>> To fix the problem, I had to make the following changes on lines 3562
>>>>> and 3563 in rdma.c:
>>>>> options->send_count = 32;
>>>>> options->recv_count = 32;
>>>>>
>>>>> The values were set to 128.
>>>>>
>>>>> I'll run some tests tomorrow to verify that it is working correctly.
>>>>> Assuming it does, what would be the expected side-effect of changing
>>>>> the values from 128 to 32? Will there be a decrease in performance?
>>>>>
>>>>>
>>>>> On Wed, Dec 1, 2010 at 10:07 AM, Jeremy Stout<stout.jeremy at gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> Here are the results of the test:
>>>>>> submit-1:/usr/local/glusterfs/3.1.1/var/log/glusterfs #
>>>>>> ibv_srq_pingpong
>>>>>> local address: LID 0x0002, QPN 0x000406, PSN 0x703b96, GID ::
>>>>>> local address: LID 0x0002, QPN 0x000407, PSN 0x618cc8, GID ::
>>>>>> local address: LID 0x0002, QPN 0x000408, PSN 0xd62272, GID ::
>>>>>> local address: LID 0x0002, QPN 0x000409, PSN 0x5db5d9, GID ::
>>>>>> local address: LID 0x0002, QPN 0x00040a, PSN 0xc51978, GID ::
>>>>>> local address: LID 0x0002, QPN 0x00040b, PSN 0x05fd7a, GID ::
>>>>>> local address: LID 0x0002, QPN 0x00040c, PSN 0xaa4a51, GID ::
>>>>>> local address: LID 0x0002, QPN 0x00040d, PSN 0xb7a676, GID ::
>>>>>> local address: LID 0x0002, QPN 0x00040e, PSN 0x56bde2, GID ::
>>>>>> local address: LID 0x0002, QPN 0x00040f, PSN 0xa662bc, GID ::
>>>>>> local address: LID 0x0002, QPN 0x000410, PSN 0xee27b0, GID ::
>>>>>> local address: LID 0x0002, QPN 0x000411, PSN 0x89c683, GID ::
>>>>>> local address: LID 0x0002, QPN 0x000412, PSN 0xd025b3, GID ::
>>>>>> local address: LID 0x0002, QPN 0x000413, PSN 0xcec8e4, GID ::
>>>>>> local address: LID 0x0002, QPN 0x000414, PSN 0x37e5d2, GID ::
>>>>>> local address: LID 0x0002, QPN 0x000415, PSN 0x29562e, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x000406, PSN 0x3b644e, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x000407, PSN 0x173320, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x000408, PSN 0xc105ea, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x000409, PSN 0x5e5ff1, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x00040a, PSN 0xff15b0, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x00040b, PSN 0xf0b152, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x00040c, PSN 0x4ced49, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x00040d, PSN 0x01da0e, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x00040e, PSN 0x69459a, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x00040f, PSN 0x197c14, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x000410, PSN 0xd50228, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x000411, PSN 0xbc9b9b, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x000412, PSN 0x0870eb, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x000413, PSN 0xfb1fbc, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x000414, PSN 0x3eefca, GID ::
>>>>>> remote address: LID 0x000b, QPN 0x000415, PSN 0xbd64c6, GID ::
>>>>>> 8192000 bytes in 0.01 seconds = 5917.47 Mbit/sec
>>>>>> 1000 iters in 0.01 seconds = 11.07 usec/iter
>>>>>>
>>>>>> fs-1:/usr/local/glusterfs/3.1.1/var/log/glusterfs # ibv_srq_pingpong
>>>>>> submit-1
>>>>>> local address: LID 0x000b, QPN 0x000406, PSN 0x3b644e, GID ::
>>>>>> local address: LID 0x000b, QPN 0x000407, PSN 0x173320, GID ::
>>>>>> local address: LID 0x000b, QPN 0x000408, PSN 0xc105ea, GID ::
>>>>>> local address: LID 0x000b, QPN 0x000409, PSN 0x5e5ff1, GID ::
>>>>>> local address: LID 0x000b, QPN 0x00040a, PSN 0xff15b0, GID ::
>>>>>> local address: LID 0x000b, QPN 0x00040b, PSN 0xf0b152, GID ::
>>>>>> local address: LID 0x000b, QPN 0x00040c, PSN 0x4ced49, GID ::
>>>>>> local address: LID 0x000b, QPN 0x00040d, PSN 0x01da0e, GID ::
>>>>>> local address: LID 0x000b, QPN 0x00040e, PSN 0x69459a, GID ::
>>>>>> local address: LID 0x000b, QPN 0x00040f, PSN 0x197c14, GID ::
>>>>>> local address: LID 0x000b, QPN 0x000410, PSN 0xd50228, GID ::
>>>>>> local address: LID 0x000b, QPN 0x000411, PSN 0xbc9b9b, GID ::
>>>>>> local address: LID 0x000b, QPN 0x000412, PSN 0x0870eb, GID ::
>>>>>> local address: LID 0x000b, QPN 0x000413, PSN 0xfb1fbc, GID ::
>>>>>> local address: LID 0x000b, QPN 0x000414, PSN 0x3eefca, GID ::
>>>>>> local address: LID 0x000b, QPN 0x000415, PSN 0xbd64c6, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x000406, PSN 0x703b96, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x000407, PSN 0x618cc8, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x000408, PSN 0xd62272, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x000409, PSN 0x5db5d9, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x00040a, PSN 0xc51978, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x00040b, PSN 0x05fd7a, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x00040c, PSN 0xaa4a51, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x00040d, PSN 0xb7a676, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x00040e, PSN 0x56bde2, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x00040f, PSN 0xa662bc, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x000410, PSN 0xee27b0, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x000411, PSN 0x89c683, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x000412, PSN 0xd025b3, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x000413, PSN 0xcec8e4, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x000414, PSN 0x37e5d2, GID ::
>>>>>> remote address: LID 0x0002, QPN 0x000415, PSN 0x29562e, GID ::
>>>>>> 8192000 bytes in 0.01 seconds = 7423.65 Mbit/sec
>>>>>> 1000 iters in 0.01 seconds = 8.83 usec/iter
>>>>>>
>>>>>> Based on the output, I believe it ran correctly.
>>>>>>
>>>>>> On Wed, Dec 1, 2010 at 9:51 AM, Anand Avati<anand.avati at gmail.com>
>>>>>> wrote:
>>>>>>>
>>>>>>> Can you verify that ibv_srq_pingpong works from the server where this
>>>>>>> log
>>>>>>> file is from?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Avati
>>>>>>>
>>>>>>> On Wed, Dec 1, 2010 at 7:44 PM, Jeremy Stout<stout.jeremy at gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Whenever I try to start or mount a GlusterFS 3.1.1 volume that uses
>>>>>>>> RDMA, I'm seeing the following error messages in the log file on the
>>>>>>>> server:
>>>>>>>> [2010-11-30 18:37:53.51270] I [nfs.c:652:init] nfs: NFS service
>>>>>>>> started
>>>>>>>> [2010-11-30 18:37:53.51362] W [dict.c:1204:data_to_str] dict:
>>>>>>>> @data=(nil)
>>>>>>>> [2010-11-30 18:37:53.51375] W [dict.c:1204:data_to_str] dict:
>>>>>>>> @data=(nil)
>>>>>>>> [2010-11-30 18:37:53.59628] E [rdma.c:2066:rdma_create_cq]
>>>>>>>> rpc-transport/rdma: testdir-client-0: creation of send_cq failed
>>>>>>>> [2010-11-30 18:37:53.59851] E [rdma.c:3771:rdma_get_device]
>>>>>>>> rpc-transport/rdma: testdir-client-0: could not create CQ
>>>>>>>> [2010-11-30 18:37:53.59925] E [rdma.c:3957:rdma_init]
>>>>>>>> rpc-transport/rdma: could not create rdma device for mthca0
>>>>>>>> [2010-11-30 18:37:53.60009] E [rdma.c:4789:init] testdir-client-0:
>>>>>>>> Failed to initialize IB Device
>>>>>>>> [2010-11-30 18:37:53.60030] E
>>>>>>>> [rpc-transport.c:971:rpc_transport_load]
>>>>>>>> rpc-transport: 'rdma' initialization failed
>>>>>>>>
>>>>>>>> On the client, I see:
>>>>>>>> [2010-11-30 18:43:49.653469] W [io-stats.c:1644:init] testdir:
>>>>>>>> dangling volume. check volfile
>>>>>>>> [2010-11-30 18:43:49.653573] W [dict.c:1204:data_to_str] dict:
>>>>>>>> @data=(nil)
>>>>>>>> [2010-11-30 18:43:49.653607] W [dict.c:1204:data_to_str] dict:
>>>>>>>> @data=(nil)
>>>>>>>> [2010-11-30 18:43:49.736275] E [rdma.c:2066:rdma_create_cq]
>>>>>>>> rpc-transport/rdma: testdir-client-0: creation of send_cq failed
>>>>>>>> [2010-11-30 18:43:49.736651] E [rdma.c:3771:rdma_get_device]
>>>>>>>> rpc-transport/rdma: testdir-client-0: could not create CQ
>>>>>>>> [2010-11-30 18:43:49.736689] E [rdma.c:3957:rdma_init]
>>>>>>>> rpc-transport/rdma: could not create rdma device for mthca0
>>>>>>>> [2010-11-30 18:43:49.736805] E [rdma.c:4789:init] testdir-client-0:
>>>>>>>> Failed to initialize IB Device
>>>>>>>> [2010-11-30 18:43:49.736841] E
>>>>>>>> [rpc-transport.c:971:rpc_transport_load] rpc-transport: 'rdma'
>>>>>>>> initialization failed
>>>>>>>>
>>>>>>>> This results in an unsuccessful mount.
>>>>>>>>
>>>>>>>> I created the mount using the following commands:
>>>>>>>> /usr/local/glusterfs/3.1.1/sbin/gluster volume create testdir
>>>>>>>> transport rdma submit-1:/exports
>>>>>>>> /usr/local/glusterfs/3.1.1/sbin/gluster volume start testdir
>>>>>>>>
>>>>>>>> To mount the directory, I use:
>>>>>>>> mount -t glusterfs submit-1:/testdir /mnt/glusterfs
>>>>>>>>
>>>>>>>> I don't think it is an Infiniband problem since GlusterFS 3.0.6 and
>>>>>>>> GlusterFS 3.1.0 worked on the same systems. For GlusterFS 3.1.0, the
>>>>>>>> commands listed above produced no error messages.
>>>>>>>>
>>>>>>>> If anyone can provide help with debugging these error messages, it
>>>>>>>> would be appreciated.
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list