[Gluster-devel] performance issue

[Matt Drew]

The CPU usage issue turned out to be a wrong - I found that someone
else was testing writes as I was working reads and the webserver.
There were no lk() calls in either the debug logs or the trace file
(probably due to locks being disabled while I was tracing).  I've now
removed the locks and io-threads translators on the server.  Current
testing environment:

brick-ns
brick1
brick2
server

brick-ns
brick1
brick2
unify

I've discovered that the server on glusterfs is performing similarly
to an older PPC OS X machine we use for testing.  I'm now searching
for similarities to see if I can identify something in common that
could cause the slowdown.

On Jan 17, 2008 9:56 PM, Anand Avati <avati at zresearch.com> wrote:
> Matt,
>  replies inline -
>
>
> >
> > I've removed all the performance translators on the client.  The
> > server has io-threads just below the server export layer, and I
> > replaced the lock translators with debug translators.  So my stack now
> > looks like this:
> >
> > bricks
> > traces
> > io-threads
> > server
> >
> > client bricks
> > unify (self-heal turned back on)
> >
> > I had checked the server, but not thoroughly.  When our scripts are
> > called, the glusterfsd CPU usage shoots up - 50-60% of one cpu
> > (Opteron 1210 dual-core).  This is much higher than the client (which
> > is usually 4-10% CPU during the same period).
>
> Is this observed after load the trace translator? trace can eat a lot of cpu
> cycles as it sets the overall logging level to DEBUG.
>
>
> >  The peak coincides with
> > the slow execution, and then immediately drops off - even though there
> > is quite a bit of I/O going still.  I see a similar spike on writes,
> > sustained for large files.  the debug trace (just above the server
> > brick) contains enormous numbers of inode creations, activations,
> > passivations, and destroys., but nothing that stands out as broken.
> > Still, the high CPU usage seems odd, especially on reads - doesn't
> > seem like there should be that much activity.
>
> can you grep through the log file and see any references of lk() calls
> happening? if so, can you retry your runs by removing posix-locks
> translator? (for the sake of diagnosis)
>
> thanks,
> avati
>