[Gluster-users] 1/4 glusterfsd's runs amok; performance suffers;

[Brian Candler]

On Sat, Aug 11, 2012 at 08:31:51AM -0700, Harry Mangalam wrote:
>    Re the size difference, I'll explicitly rebalance the brick after the
>    fix-layout finishes, but I'm even more worried about this fantastic
>    increase in CPU usage and its effect on user performance.

This presumably means you were originally running the cluster with fewer
nodes, and then added some later?

>    In the fix-layout routines (still running), I've seen CPU usage of
>    glusterfsd rise to ~400% and loadavg go up to >15 on all the servers
>    (except the pbs3, the one that originally had that problem).  That high
>    load does not last long tho (maybe a few mintes - we've just installed
>    nagios on these nodes and I'm getting a ton of emails about load
>    increasing and then decreasing on all the nodes (except pbs3).  When
>    the load goes very high on a server node, the user-end performance
>    drops appreciably.

Maybe worth trying an strace (strace -f -p <pid> 2>strace.out) on the
glusterfsd process, or whatever it is which is causing the high load, during
such a burst, just for a few seconds. The output might give some clues.