[Gluster-devel] Re: High CPU usage -- any way to reduce cpu load?

[billy cokalopolous]

A follow up. Gluster seems to do great with transfers of files (read /
write), but runs into performance issues when performing computations on the
files.

Just like mysql server causing glusterfs & glusterfsd high cpu load when
mounted on glusterfs, 5 concurrent md5sum operations on a 300MB file cause
the same high cpu usage for glusterfs and glusterfsd (see below)

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12868 root      15   0 22344 2964  904 S   32  0.1   1:06.01glusterfs
13622 root      17   0  3924 1624  512 R   30  0.1   0:00.91md5sum
12329 root      15   0  228m  81m  892 S   28  4.0   1:21.72glusterfsd
13621 root      16   0  3924 1620  512 D   27  0.1   0:01.02md5sum
13623 root      17   0  3924 1620  512 R   20  0.1   0:00.61md5sum
13620 root      16   0  3928 1628  512 R   19  0.1   0:00.89md5sum
13619 root      16   0  3928 1624  512 R   17  0.1   0:00.83md5sum
13618 root      16   0  3924 1620  512 R   17  0.1   0:00.98md5sum

Also, you can see the mysql stats here -- mysql can only utilize 7% of cpu,
but when not run on gluster, mysql utilization for SAME query is 50%.

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
12868 root      15   0 21316 1936  904 S   55  0.1   1:10.97glusterfs
12329 root      15   0  218m  63m  892 S   54  3.1   1:26.42glusterfsd
13534 mysql     15   0 44360 9340 3976 S    7  0.5   0:03.90mysqld

Is this expected behavior? Is there a way to tune to reduce
glusterfs/glusterfsd overhead? Thank you.

On Fri, Feb 29, 2008 at 9:21 AM, billy cokalopolous <
billy.cokalopolous at gmail.com> wrote:

> I'm currently testing a 1 node architecture as I was having problems with
> afr and multinode performance.
>
> write and read performance to glusterfs (single node) is now good, but
> when running mysql on top of glusterfs, performance drops dramatically due
> to high cpu usage.
>
> mysql is mounted to the glusterfs & glusterfsd is running on the same
> node.
>
> When running a simple query accessing one table, the cpu load from
> glusterfs / glusterfsd in top are relatively low.
>
> When running a 3 table join (18 files must be read, myi,myd,frm for each
> table, 2 tables per virtual table, so 3*3*2), the glusterfs/glusterfsd cpu
> load skyrockets stealing the cpu away from mysql processing.
>
> So this query on the local fs (ext3) takes 4 seconds, while the single
> node glusterfs takes 22 seconds.
>
> I've adjusted iothreads, iocache, readahead, etc,etc. None of these
> options make a difference in the cpu load. Tried using posix-locks and then
> posix.
>
> I wonder if glusterfs and mysql are stepping on each other's toes to cause
> this high load? Might the issue lie in tuning the file system & kernel
> params? Or is this cpu usage expected load?
>
> Thanks.
>
>