[Gluster-devel] Question about compile performance over GlusterFS

[Amar S. Tumballi]

Hi Craig,
 I will be looking into this issue. Btw, is there any reason you are not
using ib-verbs? instead using ib-sdp?
 Let me get back to you regarding this. Give me few days.

Regards,
Amar

On Thu, Mar 13, 2008 at 8:41 AM, Craig Tierney <Craig.Tierney at noaa.gov>
wrote:

> Amar S. Tumballi wrote:
> > Hi Craig,
> >  Thanks for a nice comparison  between GlusterFS and other network file
> > systems. But sure, before concluding about the performance, I would
> suggest
> > few improvement to your GlusterFS setup.
> >
> > 1. Try with client/protocol instead of having unify with only one
> subvolume.
> > (Unify makes sense when you have more than one subvolume, but when there
> is
> > only one subvolume, its a extra layer which may count as overhead),
> below
> > io-thread volume.
> >
> > 2. in io-cache on client side, as during kernel compile lot of *.h files
> are
> > re-read, you can give preference to *h files only.
> >
> > volume ioc
> >   type performance/io-cache
> >   subvolumes wb
> >   option priority *.h:100
> > end-volume
> >
>
> I changed the io-cache settings to those above and eliminated the use of
> the Unify
> subvolume (my scripts generate server/client configs automatically, and in
> most
> cases multiple servers are used, in this case they weren't).    The
> compile time
> went down, but not by much.  The latest test finished in 1042 seconds.
>
> What I didn't test this time is the compile directly on the storage that
> is exported
> by Gluster.  The runtime there is 399 seconds, so the underlying
> filesystem is fast.
>
> I am not making any conclusions about the performance based on these
> numbers.
> Things are going great so far, and this should be a solveable problem
> based on the
> other performance characteristics I have seen.
>
> Craig
>
>
>
> > Regards,
> > Amar
> >
> > On Wed, Mar 12, 2008 at 3:55 PM, Craig Tierney <Craig.Tierney at noaa.gov>
> > wrote:
> >
> >> I have been testing out my GlusterFS setup.  I have been
> >> very happy with the streaming IO performance and scalability.
> >> We have some users on the system now and they are seeing
> >> very good performance (fast and consistent) as compared
> >> to our other filesystem.
> >>
> >> I have a test that I created that tries to measure metadata
> >> performance by building the linux kernel.  What I have
> >> found is that GlusterFS is slower than local disk, NFS,
> >> and Panasas.  The compile time on those three systems
> >> is roughly 500 seconds.  For GlusterFS (1.3.7), the
> >> compile time is roughly 1200 seconds.  My GlusterFS filesystem
> >> is using ramdisks on the servers and communicating using
> >> IB-Verbs.  My server and client configs are below.
> >>
> >> Note I did not implement both write-behind and not read-behind
> >> based on some benchmarks I saw on the list on how it affects
> >> re-write.
> >>
> >> So, is this just because mmap isn't (yet) supported in FUSE?
> >> Or, is there something else I should be looking at.
> >>
> >> Thanks,
> >> Craig
> >>
> >>
> >> server.cfg
> >> ----------
> >>
> >> volume brick
> >>   type storage/posix                   # POSIX FS translator
> >>   option directory /tmp/scratch/export        # Export this directory
> >> end-volume
> >>
> >> volume server
> >>   type protocol/server
> >>   subvolumes brick
> >>   option transport-type ib-sdp/server     # For TCP/IP transport
> >>   option auth.ip.brick.allow *
> >> end-volume
> >>
> >> client.cfgvolume client-ns
> >>   type protocol/client
> >>   option transport-type ib-sdp/client
> >>   option remote-host w8-ib0
> >>   option remote-subvolume brick-ns
> >> end-volume
> >>
> >>
> >>
> >> volume client-w8
> >>   type protocol/client
> >>   option transport-type ib-sdp/client
> >>   option remote-host w8-ib0
> >>   option remote-subvolume brick
> >> end-volume
> >>
> >> volume unify
> >>         type cluster/unify
> >>         subvolumes  client-w8
> >>         option namespace client-ns
> >>         option scheduler rr
> >> end-volume
> >>
> >> volume iot
> >>         type performance/io-threads
> >>         subvolumes unify
> >>         option thread-count 4
> >> end-volume
> >>
> >> volume wb
> >>         type performance/write-behind
> >>         subvolumes iot
> >> end-volume
> >>
> >> volume ioc
> >>         type performance/io-cache
> >>         subvolumes wb
> >> end-volume
> >>
> >> ----------
> >>
> >>
> >>
> >>
> >> --
> >> Craig Tierney (craig.tierney at noaa.gov)
> >>
> >>
> >
> >
>
>
> --
> Craig Tierney (craig.tierney at noaa.gov)
>
>


-- 
Amar Tumballi
Gluster/GlusterFS Hacker
[bulde on #gluster/irc.gnu.org]
http://www.zresearch.com - Commoditizing Supercomputing and Superstorage!