[Gluster-users] Poor performance with AFR

[Keith Freedman]

I had a similar problem.  although not as severe as yours.
Mine was mostly involving a very large directory 
which seemed to get sent back and forth constantly.

1.4 pre5 is running now and it seems much 
improved.  My guess is that there are a number of 
improvements in 1.4 that will solve some of the issues you're facing.

I'd try to benchmark with that and see if your results improve.

At 07:31 AM 9/17/2008, Harald Stürzebecher wrote:
>Hello!
>
>First: I'm no expert for GlustersFS, I just did some testing in the
>last few weeks.
>
>2008/9/17 Stefan Boresch <stefan at mdy.univie.ac.at>:
> > I have just finished my first steps with glusterfs. Realizing in
> > principle what I wanted to do (including installation from source) was
> > astonishingly easy; however, the performance is extremely poor. Thus,
> > I'd appreciate comments and suggestions what to do/try next.
>
>[...]
>
> > The setup I have described is a first test for the eventual migration
> > of NFS mounted home dirs to glusterfs in order to enhance data safety
> > and availability (hence AFR). My problem is that for two typical use
> > scenarios, the performance is not acceptable. My two "testcases" are a
> > cp -a of a source tree including git repository of appr. 160MB from
> > the local disk to the glusterfs filesystem ("CP-A") and a make
> > statement in this source tree (in the remote glusterfs filesystem)
> > when everything is up to date ("MAKE"); i.e., after make has checked
> > all 800 or so source files it tells you that there is nothing to do. I
> > am afraid my timings speak for themselves:
> >
> >                 CP-A                  MAKE
> > local disk      < 5sec                < 0.3sec
> > NFS (100MBit)    55sec+-2sec          < 2sec
> > glusterfs (I)    4m29sec               17sec
> > glusterfs (II)   4m05sec               18sec
>
>160MB/5s = 32MB/s
>160MB/55s = 2,9MB/s
>160MB/250s = 655kB/s
>your right, there is something wrong, that's a bit slow
>
>IIRC GlusterFS has a high latency when accessing files/metadata.
>The new binary protocol to be introduced in the 
>1.4 release should improve that.
>(http://www.gluster.org/docs/index.php/GlusterFS_Roadmap#GlusterFS_1.4_-_Small_File_Performance)
>I'd suggest trying the test release of version 1.4 or waiting a month
>or two for the new stable release and then try again.
>
> > (I) unpatched fuse.ko, (II) patched fuse.ko, both over the same network
> > as NFS. Note that in (II)  the modified kernel module was used
> > with the default (distribution provided) libfuse library.
> >
> > Measurements are fairly reproducible, i.e., variations are at most +-
> > a few seconds. As you can see from the volume specs below, I tried
> > some performance options, but the timings, if anything, got worse. Based on
> > the above timings, I also don't think that the patched fuse.ko is worth
> > the pain.
> >
> > Some questions: (1) Would server side AFR 
> improve things? The servers can/could
> > talk over dedicated GBit Ethernet with 
> jumboframes. Judging from the howtos,
> > server side AFR is much more problematic 
> concerning (redundant) availibility of the
> > service, but I first *have* to get 
> performance somewhere near the NFS levels before
> > there is any sense in continuing.
>
>I'm not sure if the numbers are comparable. On write, Client side AFR
>has to send the same data twice over the network to get it to both of
>the two servers. On a slow network that has to have a very big impact
>on performance.
>
>Would the setup described in
>http://www.gluster.org/docs/index.php/NFS_Like_Standalone_Storage_Server
>be comparable to your current NFS setup?
>Did you try anything like that?
>
>Other suggested reading:
>http://www.gluster.org/docs/index.php/GlusterFS_1.3_P2P_Cluster_with_Auto_Healing
>http://www.gluster.org/docs/index.php/AFR_(Automatic_File_Replication)_-_Things_to_keep_in_mind_and_gotchas
>There might be some information for what you are trying to do.
>
>IMHO, doing server-side AFR with automatic IP-Failover and some sort
>of loadbalancing (e.g. telling half of the clients to use one server,
>telling the other half to use the second one) might be a setup better
>suited to your network (slow connection to clients, fast connection
>between servers) without compromising on redundancy.
>
> > I should maybe add that the single client scenario is not realistic; the
> > tests just reflect typical activities of my users and myself. The setup
> > eventually should serve 8-10 users with homedirs of 10-50GB; some users
> > often move GBs of data through their
> > homedirs. This is / has not been without pain using NFS, but
> > performance was acceptable so far. So even if I get my test scenario
> > up to speed, would glusterfs be up to the real task ?
>
>On a "real" network it should be:
>http://www.gluster.org/docs/index.php/GlusterFS_1.2.1-BENKI_Aggregated_I/O_vs_NFSv4_Benchmark
>:-)
>I just can't image who can afford that kind of 
>setup in a real world scenario...
>
>Harald Stürzebecher
>
>_______________________________________________
>Gluster-users mailing list
>Gluster-users at gluster.org
>http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users