[Gluster-devel] transport/ib-verbs: server: could not create SRQ

[Mickey Mazarick]

I'm going to answer my own question in case someone else wants to use 
the latest open-ib code with gluster (has port bonding and some other 
features I plan to try later).

SRQ is implemented in ofed 1.2 but needs the latest firmware from most 
vendors to work. I was cursed with an off-brand ib card but it had a 
mellanox chipset. After some tweaking we were able to patch the firmware 
with a reference firmware from the mellanox site.
http://www.mellanox.com/support/firmware_table.php

After that everything went smoothly.

Thanks!


Mickey Mazarick wrote:
> I didn't have that exact option, but I did have 
> ibv-recv-work-request-count set to 32. I also added the 
> ib-verbs-work-request-recv-count option as well. I've tried several 
> values there but I still get the same error. I'm running gluter as 
> root on a Centos 5 box upgraded to kernel 2.6.20
> I actually found it here: 
> https://wiki.openfabrics.org/tiki-index.php?page=Downloading+Code+From+the+OFA+git+Repositories 
>
> git link: git://git.openfabrics.org/~vlad/ofed_1_2/.git 
> <git://git.openfabrics.org/%7Evlad/ofed_1_2/.git>
>
> pre-patched with the latest infiniband code from openfabrics.
> below is the volume spec I'm using as well.
>
> Thanks for your help :-)
>
> -Mic
>
> ### File: /etc/glusterfs-server.vol - GlusterFS Server Volume 
> Specification
>
> volume main
>  type storage/posix
>  option directory /mnt/gluster/main
>  option debug off
> end-volume
>
> #volume main-io    #iothreads can give performance a boost
> #   type performance/io-threads
> #   option thread-count 8
> #   subvolumes main
> #end-volume
>
> volume server
>  type protocol/server
> #  option transport-type tcp/server     # For TCP/IP transport
> # option transport-type ib-sdp/server  # For Infiniband transport
> option transport-type ib-verbs/server # For Infiniband Verbs transport
> option ibv-recv-work-request-size   131072  # Higher performance if 
> its equal t     o read-ahead size
> option ibv-recv-work-request-count  32
> option ibv-send-work-request-size   131072  # Higher performance if 
> its equal t     o write-behind size
> option ibv-send-work-request-count  32
> option ib-verbs-work-request-recv-count 32
>
> #  subvolumes main-io
> subvolumes main
>  option auth.ip.main.allow 192.168.*
> end-volume
>
> ### END FILE ###
>
>
>
> Anand Avati wrote:
>> Mickey,
>>  what do you have as 'option ib-verbs-work-request-recv-count 
>> <count>' in the server spec file? Also, are you trying to run 
>> glusterfsd as non root? which is your distro?
>>
>> thanks,
>> avati
>>
>> 2007/8/6, Mickey Mazarick <mic at digitaltadpole.com 
>> <mailto:mic at digitaltadpole.com>>:
>>
>>     I'm having trouble getting the ibverbs transport working. I'm new
>>     to the
>>     infiniband world but the ib network is working correctly. I'm
>>     using OFED
>>     ver 1.2 but all the docs are for version 1.1.
>>
>>     The error in the gluster logs is:
>>     [ib-verbs.c:1285:ib_verbs_get_device] transport/ib-verbs: server:
>>     could
>>     not create SRQ
>>
>>     Can someone point me in the right direction?
>>
>>     Thanks1
>>     -Mickey Mazarick
>>
>>
>>
>>     _______________________________________________
>>     Gluster-devel mailing list
>>     Gluster-devel at nongnu.org <mailto:Gluster-devel at nongnu.org>
>>     http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>     <http://lists.nongnu.org/mailman/listinfo/gluster-devel>
>>
>>
>>
>>
>> -- 
>> It always takes longer than you expect, even when you take into 
>> account Hofstadter's Law.
>>
>> -- Hofstadter's Law 
>
>


--