[Gluster-users] Poor performance with AFR

[Stefan Boresch]

Well, here you go. As an aside, one thing I noted in my limited tests is that
raw network/bus/disk performance makes a huge difference; I was surprised
about some of the differences I observed in NFS performance between diff.
pairs of clients / servers ...

All the commented out stuff reflects things I tried before; the last
number was obtained with the files as shown. In my limited testing,
none of the performance options helped; neither did using the custom
fuse.ko

HTH,

Stefan

---------------------------------
Server:

# the physical data space
##volume brick # watch out
volume gfs
  type storage/posix
  option directory /data/export
end-volume

## the actual exported volume
#volume gfs
#  type performance/io-threads
#  option thread-count 8
#  option cache-size 64MB
#  subvolumes brick
#end-volume

# server declaration
volume server
  type protocol/server
  subvolumes gfs
  option transport-type tcp/server # For TCP/IP transport
  option auth.ip.gfs.allow *
end-volume

---------------------------------
client

volume brick1
    type protocol/client
    option transport-type tcp/client # for TCP/IP transport
    option remote-host x.y.z.w # IP address of omega
    option remote-subvolume gfs    # name of the remote volume on omega
end-volume

#;#volume brick2
#;#    type protocol/client
#;#    option transport-type tcp/client # for TCP/IP transport
#;#    option remote-host x.y.z.v # IP address of sigma
#;#    option remote-subvolume gfs    # name of the remote volume on sigma
#;#end-volume
#;#
#;#volume afr
#;#   type cluster/afr
#;#   subvolumes brick1 brick2
#;#end-volume

## performance block for cluster                   # optional!
#volume writeback
#  type performance/write-behind
#  option aggregate-size 131072
#  subvolumes afr
#end-volume

## performance block for cluster                   # optional!
#volume readahead
#  type performance/read-ahead
#  option page-size 65536
#  option page-count 16
#  subvolumes writeback
#end-volume

On Fri, Sep 19, 2008 at 01:07:08AM +0530, Chandranshu . wrote:
> Hi Stefan,
> 
> We have been trying to optimize Glusterfs on our network and our results
> match your previous results. After reading your post, I tried removing AFR
> but our performances are still poor as compared to NFS (particulalrly for
> small files). Could you please post your new volume spec file and/or any
> other improvements that you made to any of the layers? That'll be very
> helpful.
> 
> Thanks and regards
> Chandranshu
> 
> 
> > Date: Thu, 18 Sep 2008 10:47:22 +0200
> > From: stefan at mdy.univie.ac.at (Stefan Boresch)
> > Subject: Re: [Gluster-users] Poor performance with AFR
> > To: gluster-users at gluster.org
> > Message-ID: <20080918084722.GH9860 at loop.mdy.univie.ac.at>
> > Content-Type: text/plain; charset=us-ascii
> >
> > Dear everyone,
> >
> > thank you for your replies. I have some additional data that have
> > clarified issues for me a bit:
> >
> > I have repeated my tests without AFR (basically replicating the plain
> > NFS setup)
> >
> >
> > > >                 CP-A                  MAKE
> > > > local disk      < 5sec                < 0.3sec
> > > > NFS (100MBit)    55sec+-2sec          < 2sec
> > > > glusterfs (I)    4m29sec               17sec
> > > > glusterfs (II)   4m05sec               18sec
> > glusterfs w/o AFR    45+-2 sec              9sec   <==NEW
> >
> > So, most of the poor performance is due to AFR. Note that the
> > copy actually is now faster than NFS. Interestingly, make
> > still runs much slower (although compared to the actual
> > compile time, this overhead should be negligible in practice)
> >
> > Also, upon running some NFS benchmarks between the two servers, I
> > noted some strange results, letting me suspect some creeping hardware
> > issues.
> >
> > So, I guess I'll (a) wait for glusterfs 1.4.x and (b) look out for
> > some better hardware to test things in the meantime.
> >
> > Sorry if I caused confusion; I should have checked some of these things
> > earlier.
> >
> > Best regards,
> >
> > Stefan Boresch
> >
> > --
> > Stefan Boresch
> > Institute for Computational Biological Chemistry
> > University of Vienna, Waehringerstr. 17       A-1090 Vienna, Austria
> > Phone: -43-1-427752715                        Fax:   -43-1-427752790
> >
> >

-- 
Stefan Boresch
Institute for Computational Biological Chemistry
University of Vienna, Waehringerstr. 17       A-1090 Vienna, Austria
Phone: -43-1-427752715                        Fax:   -43-1-427752790