[Gluster-users] Storage really slow on k8s
Rene Bon Ciric
renich at cloudsigma.com
Wed Mar 18 17:41:25 UTC 2020
Hello,
I have a converged setup; as you suggest in
https://github.com/gluster/gluster-kubernetes.
It's a pretty standard installation. 3 nodes are dedicated. Each one
with a system + data drives. The data drive is 500 GiB.
I am suffering, for quite a while now, of extreme slowness. Here're
some network tests.
root at k8s2:~# iperf3 -Vc glusterfs0
iperf 3.1.3
Linux k8s2.test.gva.cloudsigma.com 4.15.0-72-generic #81-Ubuntu SMP
Tue Nov 26 12:20:02 UTC 2019 x86_64
Time: Wed, 11 Mar 2020 01:47:45 GMT
Connecting to host glusterfs0, port 5201
Cookie: k8s2.test.gva.cloudsigma.com.1583891
TCP MSS: 1448 (default)
[ 4] local 10.0.77.3 port 53642 connected to 10.0.77.33 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting
0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.02 GBytes 8.77 Gbits/sec 1459 3.15 MBytes
[ 4] 1.00-2.00 sec 1022 MBytes 8.58 Gbits/sec 2284 3.15 MBytes
[ 4] 2.00-3.00 sec 1005 MBytes 8.43 Gbits/sec 2230 3.15 MBytes
[ 4] 3.00-4.00 sec 982 MBytes 8.24 Gbits/sec 2334 2.38 MBytes
[ 4] 4.00-5.00 sec 988 MBytes 8.29 Gbits/sec 668 2.65 MBytes
[ 4] 5.00-6.00 sec 983 MBytes 8.25 Gbits/sec 1370 2.86 MBytes
[ 4] 6.00-7.00 sec 1010 MBytes 8.47 Gbits/sec 1017 2.12 MBytes
[ 4] 7.00-8.00 sec 995 MBytes 8.35 Gbits/sec 2220 1.27 MBytes
[ 4] 8.00-9.00 sec 923 MBytes 7.75 Gbits/sec 324 1.64 MBytes
[ 4] 9.00-10.00 sec 1001 MBytes 8.40 Gbits/sec 532 1.96 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 9.72 GBytes 8.35 Gbits/sec 14438 sender
[ 4] 0.00-10.00 sec 9.72 GBytes 8.35 Gbits/sec receiver
CPU Utilization: local/sender 28.6% (0.5%u/28.0%s), remote/receiver
22.5% (1.1%u/21.4%s)
iperf Done.
root at k8s2:~# iperf3 -Vc glusterfs1
iperf 3.1.3
Linux k8s2.test.gva.cloudsigma.com 4.15.0-72-generic #81-Ubuntu SMP
Tue Nov 26 12:20:02 UTC 2019 x86_64
Time: Wed, 11 Mar 2020 01:49:10 GMT
Connecting to host glusterfs1, port 5201
Cookie: k8s2.test.gva.cloudsigma.com.1583891
TCP MSS: 1448 (default)
[ 4] local 10.0.77.3 port 42554 connected to 10.0.77.34 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting
0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 941 MBytes 7.89 Gbits/sec 4309 2.03 MBytes
[ 4] 1.00-2.00 sec 1017 MBytes 8.53 Gbits/sec 2577 2.21 MBytes
[ 4] 2.00-3.03 sec 962 MBytes 7.83 Gbits/sec 943 2.44 MBytes
[ 4] 3.03-4.00 sec 965 MBytes 8.35 Gbits/sec 2441 2.63 MBytes
[ 4] 4.00-5.00 sec 951 MBytes 7.98 Gbits/sec 1785 2.86 MBytes
[ 4] 5.00-6.00 sec 900 MBytes 7.55 Gbits/sec 1199 3.00 MBytes
[ 4] 6.00-7.00 sec 988 MBytes 8.28 Gbits/sec 888 100 KBytes
[ 4] 7.00-8.00 sec 892 MBytes 7.49 Gbits/sec 2382 2.34 MBytes
[ 4] 8.00-9.00 sec 967 MBytes 8.11 Gbits/sec 808 2.61 MBytes
[ 4] 9.00-10.00 sec 979 MBytes 8.21 Gbits/sec 3070 2.76 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 9.34 GBytes 8.02 Gbits/sec 20402 sender
[ 4] 0.00-10.00 sec 9.34 GBytes 8.02 Gbits/sec receiver
CPU Utilization: local/sender 27.4% (0.5%u/26.9%s), remote/receiver
24.7% (0.7%u/23.9%s)
iperf Done.
root at k8s2:~# iperf3 -Vc glusterfs2
iperf 3.1.3
Linux k8s2.test.gva.cloudsigma.com 4.15.0-72-generic #81-Ubuntu SMP
Tue Nov 26 12:20:02 UTC 2019 x86_64
Time: Wed, 11 Mar 2020 02:57:33 GMT
Connecting to host glusterfs2, port 5201
Cookie: k8s2.test.gva.cloudsigma.com.1583895
TCP MSS: 1448 (default)
[ 4] local 10.0.77.3 port 59230 connected to 10.0.77.35 port 5201
Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting
0 seconds, 10 second test
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1020 MBytes 8.56 Gbits/sec 1423 1.96 MBytes
[ 4] 1.00-2.00 sec 997 MBytes 8.36 Gbits/sec 2377 2.28 MBytes
[ 4] 2.00-3.00 sec 1022 MBytes 8.58 Gbits/sec 1409 2.56 MBytes
[ 4] 3.00-4.00 sec 976 MBytes 8.19 Gbits/sec 1184 2.79 MBytes
[ 4] 4.00-5.00 sec 1011 MBytes 8.48 Gbits/sec 1063 3.00 MBytes
[ 4] 5.00-6.00 sec 1016 MBytes 8.52 Gbits/sec 2608 3.00 MBytes
[ 4] 6.00-7.00 sec 979 MBytes 8.21 Gbits/sec 1337 1.73 MBytes
[ 4] 7.00-8.00 sec 1011 MBytes 8.48 Gbits/sec 746 2.11 MBytes
[ 4] 8.00-9.00 sec 1013 MBytes 8.50 Gbits/sec 2190 2.43 MBytes
[ 4] 9.00-10.00 sec 930 MBytes 7.80 Gbits/sec 1284 2.67 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
Test Complete. Summary Results:
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 9.74 GBytes 8.37 Gbits/sec 15621 sender
[ 4] 0.00-10.00 sec 9.74 GBytes 8.37 Gbits/sec receiver
CPU Utilization: local/sender 26.5% (0.4%u/26.0%s), remote/receiver
30.1% (1.2%u/28.9%s)
iperf Done.
And some dd tests:
/mnt # dd if=/dev/zero of=file.img bs=4k count=1M
^C2082+0 records in
2082+0 records out
8527872 bytes (8.5 MB, 8.1 MiB) copied, 178.228 s, 47.8 kB/s
So, to me, it seems that something is completely wrong here and I
dunno why. I'd appreciate a little help. Installations are timing out
and stuff.
Any performance tips would be appreciated.
--
Renich Bon Ćirić
Senior SysAdmin
CloudSigma AG
====================
This email is from CLOUDSIGMA AG. The contents of this email and any
attachments are confidential to the intended recipient. They may not
be disclosed to or used by or copied in any way by anyone other than
the intended recipient. If this email is received in error, please
contact CLOUDSIGMA AG on +41 (0)44 585 39 07 quoting the name of the
sender and the email address to which it has been sent and then delete
it. Please note that neither CLOUDSIGMA AG nor the sender accepts any
responsibility for viruses and it is your responsibility to scan or
otherwise check this email and any attachments. CLOUDSIGMA AG is a
limited company registered in Canton Zürich, Switzerland (registered
number CHE-115.204.434) with registered offices at Badenerstrasse
549, 8048 Zurich, Switzerland. For further information, please refer
to www.cloudsigma.com .
More information about the Gluster-users
mailing list