[Gluster-users] How to find out what GlusterFS is doing

[mabi]

Hello,

I have a 3 node replica including arbiter GlusterFS 7.8 server with 3 volumes and the two nodes (not arbiter) seem to have a high load due to the glusterfsd brick process taking all CPU resources (12 cores).

Checking these two servers with iostat command shows that the disks are not so busy and that they are mostly doing writes activity. On the FUSE clients there is not so much activity so I was wondering how to find out or explain why GlusterFS is currently generating such a high load on these two servers (the arbiter does not show any high load). There are no files currently healing either. This volume is the only volume which has the quota enabled if this might be a hint. So does anyone know how to see why GlusterFS is so busy on a specific volume?

Here is a sample "vmstat 60" of one of the nodes:

onadmin at gfs1b:~$ vmstat 60
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 9  2      0 22296776  32004 260284    0    0    33   301  153   39  2 60 36  2  0
13  0      0 22244540  32048 260456    0    0   343  2798 10898 367652  2 80 16  1  0
18  0      0 22215740  32056 260672    0    0   308  2524 9892 334537  2 83 14  1  0
18  0      0 22179348  32084 260828    0    0   169  2038 8703 250351  1 88 10  0  0

I already tried rebooting but that did not help and there is nothing special in the log files either.

Best regards,
Mabi