[Gluster-devel] Profiling glusterfs (Re: Mirrored GlusterFS -- very poor read performance)

[Mikhail T.]

On 03.01.2014 01:59, Mikhail T. wrote:
> We expected to pay /some/ performance penalty for the features, but
> the actual numbers are causing a sticker-shock...

We tried to mount GlusterFS replicated volume using NFS, rather than
glusterfs/fuse and were pleasantly surprised. The read-performance
improved dramatically -- and is in line with that of the NetApp
NFS-server we are also using:


	Local FS
	NetApp NFS 	GlusterFS 	GlusterNFS

	Average 	Minimum 	Average 	Minimum 	Average 	Minimum 	Average 	Minimum
Small static file 	1.532 	0.513 	4.027 	0.916 	27.81 	7.184 	5.591 	1.394
Large static file 	14.45 	2.721 	14.56 	3.643 	37.90 	7.433 	14.95 	4.829

This suggests, the performance problem is not on the Gluster-servers,
but on the client-side and I tried to use the profiler. Although the
software is not easily profilable due to its use of shared libraries
<https://bugzilla.redhat.com/show_bug.cgi?id=762856>, partial coverage
is possible... Here is what I did:

    % env LDFLAGS="-pg" CFLAGS="-pg -O2 -march=core2 -pipe
    -fno-strict-aliasing -g" ./configure --enable-static
    % make
    % cd glusterfsd/src
    % cc -pg -o glusterfsd.profilable *.o -lpthread
    ../../rpc/xdr/src/*.o ../../rpc/rpc-lib/src/*.o -lcrypto -lz
    ../../libglusterfs/src/*.o ../../xlators/mount/fuse/src/*.o -L
    ../../libglusterfs/src/.libs/ -lglusterfs -L
    ../../rpc/rpc-lib/src/.libs -lgfrpc -L ../../rpc/xdr/src/.libs -lgfxdr
    % ln -s glusterfsd.profiled glusterfs
    % ./glusterfs --no-daemon ......

I then ran some tests for two minutes and umounted the share. The
resulting gmon.out does not cover any calls, that took place in the
explicitly dlopen-ed objects (like fuse.so), but everything else is
included: my glusterfsd.profilable executable does not use shared
libraries for its own work, it links explicitly with the .o-files. (The
shared libs are needed only to be able to dlopen various "xlators",
which expect certain symbols to be available.)

Now, my testing did repeated requests of two files -- for one minute
each. First a tiny 430-byte file, and then a bigger 93Kb one. In 122
seconds there were about 55K file-transfers and glusterfs-process
accumulated 135 seconds of CPU-time... According to gprof, these 55K
transfers resulted in 597971 calls to each of the rpc_clnt_notify and
rpc_transport_notify, /each/ accounting for over 34% of the total time.

Joseph Landman indicated earlier
<https://lists.gnu.org/archive/html/gluster-devel/2014-01/msg00011.html>, that
he blames the kernel/user-space context-switching, and that surely is
responsible for /some/ overhead. But the bulk of it seems to be in the
client/server communication (that is, between the glusterfs client and
glusterfsd server), which appears to be unnecessarily chatty. Why are
there 100 notifications for each file-transfer? I wonder, if this can be
improved bringing the performance of glusterfs-mounts closer to that of
the NFS method... (In fact, the NFS method may benefit from such an
optimization too.)

I may be reading profiler's output wrong -- I am rusty in this area.
Would anybody else care to take a look at the results:

  * Plain <http://aldan.algebra.com/%7Emi/tmp/glusterfs-gprof.txt>
    "gprof glusterfsd.profiled"
  * Annotated source
    <http://aldan.algebra.com/%7Emi/tmp/glusterfs-annotated.txt>
    produced by "gprof -A -l -x glusterfsd.profiled"

Please, let me know, if you'd like me to massage the data in some
different way -- I'll keep the raw results around for some time... Thank
you! Yours,

    -mi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20140104/abe38312/attachment-0001.html>