[Gluster-users] RDMA connectivity not available with GlusterFS 3.5.2

[Mohammed Rafi K C]

On 10/08/2014 11:54 AM, Geoffrey Letessier wrote:
> Hello,
>
> First of all, thanks to spend some times to help me to fix my issues.
>
> Previously, some weeks ago, i’ve already installed a GlusterFS patched
> version
> (http://download.gluster.org/pub/gluster/glusterfs/nightly/glusterfs-3.5/epel-6-x86_64/glusterfs-3.5.20140922.da1657d-1.autobuild/ )
> to fix quota daemon (and libc) issues thanks to Krutika Dhananjay.

> Do this patched version you offer me is compatible with the previously
> said one? If yes, is it possible to get RPM packages?
>

   I hope it will be compatible with the quota patch. But we need to
figure out the availability of rpm packages for 3.5,  hopefully i can
let you know the availability of  RPM packages within 1 day.

> Currently we have around 42TB used disk space (per replica); so is
> there any risk to apply this patched version on a production environment?

    Currently rdma is a tech preview , which is aimed to release in next
versions. Once you applied the patch, all the communication will go
through rdma instead of tcp. I hope there will not be any major risks.

>
> In addition, concerning my global IO performances, do you feel my
> settings as optimal?

   I'm not sure about IO performance and settings of your volume,
perhaps some one from this group can help you about that.

Rafi KC
>
> Thanks in advance,
> Geoffrey
>
> Le 8 oct. 2014 à 06:58, Mohammed Rafi K C <rkavunga at redhat.com
> <mailto:rkavunga at redhat.com>> a écrit :
>
>>
>> On 10/08/2014 04:15 AM, Geoffrey Letessier wrote:
>>> Dears,
>>>
>>> I have a HPC cluster composed by 4 storage nodes (8x 24TB RAID6
>>> bricks, 2 per nodes) and 62 compute nodes, interconnected via
>>> Infiniband QDR technology.
>>>
>>> NB: each brick provide around 1.2-1.5TBs write performances.
>>>  
>>> My main volume is defined as below
>>> Volume Name: vol_home
>>> Type: Distributed-Replicate
>>> Volume ID: f6ebcfc1-b735-4a0e-b1d7-47ed2d2e7af6
>>> Status: Started
>>> Number of Bricks: 4 x 2 = 8
>>> Transport-type: tcp,rdma
>>> Bricks:
>>> Brick1: ib-storage1:/export/brick_home/brick1
>>> Brick2: ib-storage2:/export/brick_home/brick1
>>> Brick3: ib-storage3:/export/brick_home/brick1
>>> Brick4: ib-storage4:/export/brick_home/brick1
>>> Brick5: ib-storage1:/export/brick_home/brick2
>>> Brick6: ib-storage2:/export/brick_home/brick2
>>> Brick7: ib-storage3:/export/brick_home/brick2
>>> Brick8: ib-storage4:/export/brick_home/brick2
>>> Options Reconfigured:
>>> features.quota: on
>>> diagnostics.brick-log-level: CRITICAL
>>> auth.allow: localhost,127.0.0.1,10.*
>>> nfs.disable: on
>>> performance.cache-size: 64MB
>>> performance.write-behind-window-size: 1MB
>>> performance.quick-read: on
>>> performance.io-cache: on
>>> performance.io-thread-count: 64
>>> features.default-soft-limit: 90%
>>>
>>> But, in the cluster, when I try to mount my volume specifying RDMA
>>> transport type, i notice all my communication go through TCP stack
>>> (all network packet are visible on ib0 network interface with ifstat
>>> shell command), not through RDMA
>>> [root at lucifer ~]# mount -t glusterfs -o
>>> transport=rdma,direct-io-mode=disable localhost:vol_home /home
>>> [root at lucifer ~]# mount|grep vol_home.rdma
>>> localhost:vol_home.rdma on /home type fuse.glusterfs
>>> (rw,default_permissions,allow_other,max_read=131072)
>>> [root at lucifer ~]# ifstat -i ib0
>>>        ib0        
>>>  KB/s in  KB/s out
>>> 25313.60   6776.44
>>> 26258.96   9064.92
>>> 28272.97  10034.15
>>> 23495.09   8504.84
>>> 21842.41   7161.69
>>> ^C
>>>
>>> So, my best noticed throughput is around 400MBs, but basically
>>> around 200-250MBs, although I can read on the net i can expect to
>>> achieve around 800-900MBs -sometimes more- with RDMA transport type.
>>>
>>> Can anyone help me to make it work?
>>
>>     There is known issue in rdma that volume with transport type as
>> tcp,rdma will mount as tcp. the fix for the same is under review. You
>> can pull the patch from  git fetch
>> https://review.gluster.org/glusterfs refs/changes/98/8498/7 && git
>> format-patch -1  FETCH_HEAD. Applying the patch will help to mount
>> the tcp,rdma volume as rdma. If you are mounting a tcp,rdma volume as
>> RDMA fuse mount you can also append  *.rdma *with volname**instead of
>> -o option.
>>
>>
>>   Let me know your result, If possible I would like to know the
>> version of gluster you are currently using.**
>>   Rafi KC.
>>
>>>
>>> In addition, are my volume settings look like optimal?
>>>
>>> Thanks in advance,
>>> Geoffrey
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141008/077a565b/attachment.html>