[Gluster-users] Slow Replication over infiniband

[joel vennin]

Please, some one can give me clue concerning the increase of the transfer
rate, I ready to hack code if necessary.

Thank you.

On Tue, Oct 27, 2009 at 3:30 PM, joel vennin <joel.vennin at gmail.com> wrote:

> Hi, I've a setup composed with 3 nodes(zodiac1, zodiac2, zodiac3)
>
> Node config: 8 cores W5580  at 3.20GHz, 76 Go RAM, 40 To, infiniband mlx4_0
> (20 Gb)
> Gluster Version: 2.0.7
> Kernel Linux 2.6.30 / Fuse 2.7.4
>
>
> Config file:
>
> #----------------------------------------------------------
> # SERVER SIDE
> volume posix-1
>        type storage/posix
>        option directory /data/yacht-data2
> end-volume
>
> volume locks-1
>   type features/locks
>   subvolumes posix-1
> end-volume
>
>
> volume brick-1
>        type performance/io-threads
>        option thread-count 4
>        subvolumes locks-1
> end-volume
>
> volume server
>        type protocol/server
>        option transport-type                     ib-verbs
>        option transport.ib-verbs.device-name     mlx4_0
>        option auth.addr.brick-1.allow *
>        subvolumes brick-1 #brick-2
> end-volume
>
>
> # CLIENT NODE DECLARATION
>
> #
> # ZODIAC 1 CLIENT
> volume zbrick1
>        type protocol/client
>        option transport-type ib-verbs
>        option remote-host 192.168.3.200
>        option remote-subvolume brick-1
> end-volume
>
> # ZODIAC 2
> volume zbrick2
>        type protocol/client
>        option transport-type   ib-verbs
>        option remote-host      192.168.3.201
>        option remote-subvolume brick-1
> end-volume
>
> # ZODIAC 3
> volume zbrick3
>        type protocol/client
>        option transport-type   ib-verbs
>        option remote-host      192.168.3.202
>        option remote-subvolume brick-1
> end-volume
>
> # MEGA REPLICATE
> volume mega-replicator
>        type cluster/replicate
>        subvolumes zbrick1 zbrick2 zbrick3
> end-volume
> #----------------------------------------------------------
>
> Command on each server to start glusterfs:
>    glusterfs -f myreplica.vol /mnt/data/
>
>
>
> The scenario that I use:
>     zodiac1 and zodiac2 are correctly synchronized. Once every thing is ok
> for zodiac1 and zodiac2, I decide to start the zodiac3. I order to
> synchronize it, I do on the
> box ls -aLR /mnt/data. So the replication start but the transfer rate is
> really slow: around 30 MB/s !
>
> Have you an idea how can I increase this rate ?
>
>
> Thank you.
>
>
>
>
>