[Gluster-users] Poor performance with AFR

[Harald Stürzebecher]

Hello!

First: I'm no expert for GlustersFS, I just did some testing in the
last few weeks.

2008/9/17 Stefan Boresch <stefan at mdy.univie.ac.at>:
> I have just finished my first steps with glusterfs. Realizing in
> principle what I wanted to do (including installation from source) was
> astonishingly easy; however, the performance is extremely poor. Thus,
> I'd appreciate comments and suggestions what to do/try next.

[...]

> The setup I have described is a first test for the eventual migration
> of NFS mounted home dirs to glusterfs in order to enhance data safety
> and availability (hence AFR). My problem is that for two typical use
> scenarios, the performance is not acceptable. My two "testcases" are a
> cp -a of a source tree including git repository of appr. 160MB from
> the local disk to the glusterfs filesystem ("CP-A") and a make
> statement in this source tree (in the remote glusterfs filesystem)
> when everything is up to date ("MAKE"); i.e., after make has checked
> all 800 or so source files it tells you that there is nothing to do. I
> am afraid my timings speak for themselves:
>
>                 CP-A                  MAKE
> local disk      < 5sec                < 0.3sec
> NFS (100MBit)    55sec+-2sec          < 2sec
> glusterfs (I)    4m29sec               17sec
> glusterfs (II)   4m05sec               18sec

160MB/5s = 32MB/s
160MB/55s = 2,9MB/s
160MB/250s = 655kB/s
your right, there is something wrong, that's a bit slow

IIRC GlusterFS has a high latency when accessing files/metadata.
The new binary protocol to be introduced in the 1.4 release should improve that.
(http://www.gluster.org/docs/index.php/GlusterFS_Roadmap#GlusterFS_1.4_-_Small_File_Performance)
I'd suggest trying the test release of version 1.4 or waiting a month
or two for the new stable release and then try again.

> (I) unpatched fuse.ko, (II) patched fuse.ko, both over the same network
> as NFS. Note that in (II)  the modified kernel module was used
> with the default (distribution provided) libfuse library.
>
> Measurements are fairly reproducible, i.e., variations are at most +-
> a few seconds. As you can see from the volume specs below, I tried
> some performance options, but the timings, if anything, got worse. Based on
> the above timings, I also don't think that the patched fuse.ko is worth
> the pain.
>
> Some questions: (1) Would server side AFR improve things? The servers can/could
> talk over dedicated GBit Ethernet with jumboframes. Judging from the howtos,
> server side AFR is much more problematic concerning (redundant) availibility of the
> service, but I first *have* to get performance somewhere near the NFS levels before
> there is any sense in continuing.

I'm not sure if the numbers are comparable. On write, Client side AFR
has to send the same data twice over the network to get it to both of
the two servers. On a slow network that has to have a very big impact
on performance.

Would the setup described in
http://www.gluster.org/docs/index.php/NFS_Like_Standalone_Storage_Server
be comparable to your current NFS setup?
Did you try anything like that?

Other suggested reading:
http://www.gluster.org/docs/index.php/GlusterFS_1.3_P2P_Cluster_with_Auto_Healing
http://www.gluster.org/docs/index.php/AFR_(Automatic_File_Replication)_-_Things_to_keep_in_mind_and_gotchas
There might be some information for what you are trying to do.

IMHO, doing server-side AFR with automatic IP-Failover and some sort
of loadbalancing (e.g. telling half of the clients to use one server,
telling the other half to use the second one) might be a setup better
suited to your network (slow connection to clients, fast connection
between servers) without compromising on redundancy.

> I should maybe add that the single client scenario is not realistic; the
> tests just reflect typical activities of my users and myself. The setup
> eventually should serve 8-10 users with homedirs of 10-50GB; some users
> often move GBs of data through their
> homedirs. This is / has not been without pain using NFS, but
> performance was acceptable so far. So even if I get my test scenario
> up to speed, would glusterfs be up to the real task ?

On a "real" network it should be:
http://www.gluster.org/docs/index.php/GlusterFS_1.2.1-BENKI_Aggregated_I/O_vs_NFSv4_Benchmark
:-)
I just can't image who can afford that kind of setup in a real world scenario...

Harald Stürzebecher