<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 5, 2020 at 4:18 PM mabi <<a href="mailto:mabi@protonmail.ch">mabi@protonmail.ch</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Below is the top output of running "top -bHd d" on one of the nodes, maybe that can help to see what that glusterfsd process is doing?<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
4375 root 20 0 2856784 120492 8360 D 61.1 0.4 117:09.29 glfs_iotwr001<br></blockquote><div><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Waiting for IO, just like the rest of those in D state. </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">You may have a slow storage subsystem. How many cores do you have, btw?</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Y.</div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
4385 root 20 0 2856784 120492 8360 R 61.1 0.4 117:12.92 glfs_iotwr003<br>
4387 root 20 0 2856784 120492 8360 R 61.1 0.4 117:32.19 glfs_iotwr005<br>
4388 root 20 0 2856784 120492 8360 R 61.1 0.4 117:28.87 glfs_iotwr006<br>
4391 root 20 0 2856784 120492 8360 D 61.1 0.4 117:20.71 glfs_iotwr008<br>
4395 root 20 0 2856784 120492 8360 D 61.1 0.4 117:17.22 glfs_iotwr009<br>
4405 root 20 0 2856784 120492 8360 R 61.1 0.4 117:19.52 glfs_iotwr00d<br>
4406 root 20 0 2856784 120492 8360 R 61.1 0.4 117:29.51 glfs_iotwr00e<br>
4366 root 20 0 2856784 120492 8360 D 55.6 0.4 117:27.58 glfs_iotwr000<br>
4386 root 20 0 2856784 120492 8360 D 55.6 0.4 117:22.77 glfs_iotwr004<br>
4390 root 20 0 2856784 120492 8360 D 55.6 0.4 117:26.49 glfs_iotwr007<br>
4396 root 20 0 2856784 120492 8360 R 55.6 0.4 117:23.68 glfs_iotwr00a<br>
4376 root 20 0 2856784 120492 8360 D 50.0 0.4 117:36.17 glfs_iotwr002<br>
4397 root 20 0 2856784 120492 8360 D 50.0 0.4 117:11.09 glfs_iotwr00b<br>
4403 root 20 0 2856784 120492 8360 R 50.0 0.4 117:26.34 glfs_iotwr00c<br>
4408 root 20 0 2856784 120492 8360 D 50.0 0.4 117:27.47 glfs_iotwr00f<br>
9814 root 20 0 2043684 75208 8424 D 22.2 0.2 50:15.20 glfs_iotwr003<br>
28131 root 20 0 2043684 75208 8424 R 22.2 0.2 50:07.46 glfs_iotwr004<br>
2208 root 20 0 2043684 75208 8424 R 22.2 0.2 49:32.70 glfs_iotwr008<br>
2372 root 20 0 2043684 75208 8424 R 22.2 0.2 49:52.60 glfs_iotwr009<br>
2375 root 20 0 2043684 75208 8424 D 22.2 0.2 49:54.08 glfs_iotwr00c<br>
767 root 39 19 0 0 0 R 16.7 0.0 67:50.83 dbuf_evict<br>
4132 onadmin 20 0 45292 4184 3176 R 16.7 0.0 0:00.04 top<br>
28484 root 20 0 2043684 75208 8424 R 11.1 0.2 49:41.34 glfs_iotwr005<br>
2376 root 20 0 2043684 75208 8424 R 11.1 0.2 49:49.49 glfs_iotwr00d<br>
2719 root 20 0 2043684 75208 8424 R 11.1 0.2 49:58.61 glfs_iotwr00e<br>
4384 root 20 0 2856784 120492 8360 S 5.6 0.4 4:01.27 glfs_rpcrqhnd<br>
3842 root 20 0 2043684 75208 8424 S 5.6 0.2 0:30.12 glfs_epoll001<br>
1 root 20 0 57696 7340 5248 S 0.0 0.0 0:03.59 systemd<br>
2 root 20 0 0 0 0 S 0.0 0.0 0:09.57 kthreadd<br>
3 root 20 0 0 0 0 S 0.0 0.0 0:00.16 ksoftirqd/0<br>
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H<br>
7 root 20 0 0 0 0 S 0.0 0.0 0:07.36 rcu_sched<br>
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh<br>
9 root rt 0 0 0 0 S 0.0 0.0 0:00.03 migration/0<br>
10 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 lru-add-drain<br>
11 root rt 0 0 0 0 S 0.0 0.0 0:00.01 watchdog/0<br>
12 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/0<br>
13 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuhp/1<br>
<br>
Any clues anyone?<br>
<br>
The load is really high around 20 now on the two nodes...<br>
<br>
<br>
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐<br>
On Thursday, November 5, 2020 11:50 AM, mabi <<a href="mailto:mabi@protonmail.ch" target="_blank">mabi@protonmail.ch</a>> wrote:<br>
<br>
> Hello,<br>
><br>
> I have a 3 node replica including arbiter GlusterFS 7.8 server with 3 volumes and the two nodes (not arbiter) seem to have a high load due to the glusterfsd brick process taking all CPU resources (12 cores).<br>
><br>
> Checking these two servers with iostat command shows that the disks are not so busy and that they are mostly doing writes activity. On the FUSE clients there is not so much activity so I was wondering how to find out or explain why GlusterFS is currently generating such a high load on these two servers (the arbiter does not show any high load). There are no files currently healing either. This volume is the only volume which has the quota enabled if this might be a hint. So does anyone know how to see why GlusterFS is so busy on a specific volume?<br>
><br>
> Here is a sample "vmstat 60" of one of the nodes:<br>
><br>
> onadmin@gfs1b:~$ vmstat 60<br>
> procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----<br>
> r b swpd free buff cache si so bi bo in cs us sy id wa st<br>
> 9 2 0 22296776 32004 260284 0 0 33 301 153 39 2 60 36 2 0<br>
> 13 0 0 22244540 32048 260456 0 0 343 2798 10898 367652 2 80 16 1 0<br>
> 18 0 0 22215740 32056 260672 0 0 308 2524 9892 334537 2 83 14 1 0<br>
> 18 0 0 22179348 32084 260828 0 0 169 2038 8703 250351 1 88 10 0 0<br>
><br>
> I already tried rebooting but that did not help and there is nothing special in the log files either.<br>
><br>
> Best regards,<br>
> Mabi<br>
<br>
<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div></div>