[Gluster-devel] performance issue

[Anand Avati]

Matt,
 replies inline -


> I've removed all the performance translators on the client.  The
> server has io-threads just below the server export layer, and I
> replaced the lock translators with debug translators.  So my stack now
> looks like this:
>
> bricks
> traces
> io-threads
> server
>
> client bricks
> unify (self-heal turned back on)
>
> I had checked the server, but not thoroughly.  When our scripts are
> called, the glusterfsd CPU usage shoots up - 50-60% of one cpu
> (Opteron 1210 dual-core).  This is much higher than the client (which
> is usually 4-10% CPU during the same period).


Is this observed after load the trace translator? trace can eat a lot of cpu
cycles as it sets the overall logging level to DEBUG.


 The peak coincides with
> the slow execution, and then immediately drops off - even though there
> is quite a bit of I/O going still.  I see a similar spike on writes,
> sustained for large files.  the debug trace (just above the server
> brick) contains enormous numbers of inode creations, activations,
> passivations, and destroys., but nothing that stands out as broken.
> Still, the high CPU usage seems odd, especially on reads - doesn't
> seem like there should be that much activity.


can you grep through the log file and see any references of lk() calls
happening? if so, can you retry your runs by removing posix-locks
translator? (for the sake of diagnosis)

thanks,
avati