[Gluster-users] Slow Replication over infiniband

[joel vennin]

Thank you for the reply, but this kind of config change nothing concerning
the replication transfer rate.

Note that glusterfs cpu usage on all box takes less than 3% of CPU.

All the box can write data with a transfer rate at 800MB/s, but i'm still
blocked at 30MB/s

Any help/suggestion is welcome

Joel

On Thu, Oct 29, 2009 at 9:04 AM, Christian Marnitz <
christian.marnitz at icesmedia.de> wrote:

> Hi,
>
> take a look to the following performance-features:
>
> ###################################################
> ##### performance
> ###################################################
> volume iot
>  type performance/io-threads
>  option thread-count 16  # deault is 16
>  option min-threads 32
>  option max-threads 256
>  subvolumes mega-replicator
> end-volume
>
> ##### writebehind - write aggregation
> volume wb
>  type performance/write-behind
>  option cache-size 3MB         # default is equal to aggregate-size
>  option flush-behind on        # default is 'off'
>  subvolumes iot
> end-volume
>
> ##### read-ahead - read aggregation
> volume readahead
>  type performance/read-ahead
>  #option page-size 1048576      # 128kb is the default option
>  option page-count 4           # 2 is default option
>  option force-atime-update off # default is off
>  subvolumes wb
> end-volume
>
>
> ###### no chaching - disable if need realtime
> volume ioc
>  type performance/io-cache
>  option cache-size 128MB   # default is 32MB
>  option page-size 1MB      #128KB is default option
>  option cache-timeout 1        # default is 1
>  subvolumes wb
> end-volume
>
> ---
>
> Here is the info written, what happens behind:
>
>
> http://gluster.com/community/documentation/index.php/Translators/performance/readahead
>
> http://gluster.com/community/documentation/index.php/Translators/performance/writebehind
>
>
> http://gluster.com/community/documentation/index.php/Translators/performance/io-threads
>
> http://gluster.com/community/documentation/index.php/Translators/performance/io-cache
>
>
> I personally use writebehind and io-threads. It depends on your requests,
> what you need.
>
> Greetings,
> Christian
>
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: gluster-users-bounces at gluster.org [mailto:
> gluster-users-bounces at gluster.org] Im Auftrag von joel vennin
> Gesendet: Donnerstag, 29. Oktober 2009 07:44
> An: gluster-users at gluster.org
> Betreff: Re: [Gluster-users] Slow Replication over infiniband
>
> Please, some one can give me clue concerning the increase of the transfer
> rate, I ready to hack code if necessary.
>
> Thank you.
>
> On Tue, Oct 27, 2009 at 3:30 PM, joel vennin <joel.vennin at gmail.com>
> wrote:
>
> > Hi, I've a setup composed with 3 nodes(zodiac1, zodiac2, zodiac3)
> >
> > Node config: 8 cores W5580  at 3.20GHz, 76 Go RAM, 40 To, infiniband
> > mlx4_0 (20 Gb) Gluster Version: 2.0.7 Kernel Linux 2.6.30 / Fuse 2.7.4
> >
> >
> > Config file:
> >
> > #----------------------------------------------------------
> > # SERVER SIDE
> > volume posix-1
> >        type storage/posix
> >        option directory /data/yacht-data2 end-volume
> >
> > volume locks-1
> >   type features/locks
> >   subvolumes posix-1
> > end-volume
> >
> >
> > volume brick-1
> >        type performance/io-threads
> >        option thread-count 4
> >        subvolumes locks-1
> > end-volume
> >
> > volume server
> >        type protocol/server
> >        option transport-type                     ib-verbs
> >        option transport.ib-verbs.device-name     mlx4_0
> >        option auth.addr.brick-1.allow *
> >        subvolumes brick-1 #brick-2
> > end-volume
> >
> >
> > # CLIENT NODE DECLARATION
> >
> > #
> > # ZODIAC 1 CLIENT
> > volume zbrick1
> >        type protocol/client
> >        option transport-type ib-verbs
> >        option remote-host 192.168.3.200
> >        option remote-subvolume brick-1 end-volume
> >
> > # ZODIAC 2
> > volume zbrick2
> >        type protocol/client
> >        option transport-type   ib-verbs
> >        option remote-host      192.168.3.201
> >        option remote-subvolume brick-1 end-volume
> >
> > # ZODIAC 3
> > volume zbrick3
> >        type protocol/client
> >        option transport-type   ib-verbs
> >        option remote-host      192.168.3.202
> >        option remote-subvolume brick-1 end-volume
> >
> > # MEGA REPLICATE
> > volume mega-replicator
> >        type cluster/replicate
> >        subvolumes zbrick1 zbrick2 zbrick3 end-volume
> > #----------------------------------------------------------
> >
> > Command on each server to start glusterfs:
> >    glusterfs -f myreplica.vol /mnt/data/
> >
> >
> >
> > The scenario that I use:
> >     zodiac1 and zodiac2 are correctly synchronized. Once every thing
> > is ok for zodiac1 and zodiac2, I decide to start the zodiac3. I order
> > to synchronize it, I do on the box ls -aLR /mnt/data. So the
> > replication start but the transfer rate is really slow: around 30 MB/s
> > !
> >
> > Have you an idea how can I increase this rate ?
> >
> >
> > Thank you.
> >
> >
> >
> >
> >
>
>