[Gluster-users] Gluster performance issues - need advise
Strahil Nikolov
hunter86_bg at yahoo.com
Wed Jan 23 15:07:25 UTC 2019
Hello Community,
recently I have built a new lab based on oVirt and CentOS 7.
During deployment I had some hicups, but now the engine is up and running - but gluster is causing me trouble.
Symptoms: Slow VM install from DVD, poor write performance. The latter has been tested via:
dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_data bs=1M count=1000 status=progress
The reported speed is 60MB/s which is way too low for my setup.
My lab design:
https://drive.google.com/file/d/1SiW21ASPXHRAEuE_jZ50R3FoO-NcnFqT/view?usp=sharing
Gluster version is 3.12.15
So far I have done:
1. Added 'server.allow-insecure on' (with 'option rpc-auth-allow-insecure on' in glusterd.vol)
Volume info after that change:
Volume Name: data
Type: Replicate
Volume ID: 9b06a1e9-8102-4cd7-bc56-84960a1efaa2
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: ovirt1.localdomain:/gluster_bricks/data/data
Brick2: ovirt2.localdomain:/gluster_bricks/data/data
Brick3: ovirt3.localdomain:/gluster_bricks/data/data (arbiter)
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.low-prio-threads: 32
network.remote-dio: off
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
storage.owner-uid: 36
storage.owner-gid: 36
network.ping-timeout: 30
performance.strict-o-direct: on
cluster.granular-entry-heal: enable
server.allow-insecure: on
Seems no positive or negative effect so far.
2. Tested with tmpfs on all bricks -> ovirt1 mounted gluster volume -> max 60MB/s (bs=1M without 'oflag=direct')
[root at ovirt1 data]# dd if=/dev/zero of=large_io bs=1M count=4000 status=progress
4177526784 bytes (4.2 GB) copied, 70.843409 s, 59.0 MB/s
4000+0 records in
4000+0 records out
4194304000 bytes (4.2 GB) copied, 71.1407 s, 59.0 MB/s
[root at ovirt1 data]# rm -f large_io
[root at ovirt1 data]# gluster volume profile data info
Brick: ovirt1.localdomain:/gluster_bricks/data/data
---------------------------------------------------
Cumulative Stats:
Block Size: 131072b+
No. of Reads: 8
No. of Writes: 44968
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 3 FORGET
0.00 0.00 us 0.00 us 0.00 us 35 RELEASE
0.00 0.00 us 0.00 us 0.00 us 28 RELEASEDIR
0.00 78.00 us 78.00 us 78.00 us 1 FSTAT
0.00 35.67 us 26.00 us 73.00 us 6 FLUSH
0.00 324.00 us 324.00 us 324.00 us 1 XATTROP
0.00 45.80 us 38.00 us 54.00 us 10 STAT
0.00 227.67 us 216.00 us 242.00 us 3 CREATE
0.00 113.38 us 68.00 us 381.00 us 8 READ
0.00 39.82 us 1.00 us 148.00 us 28 OPENDIR
0.00 67.54 us 10.00 us 283.00 us 24 GETXATTR
0.00 59.97 us 45.00 us 113.00 us 32 OPEN
0.00 24.41 us 13.00 us 89.00 us 161 INODELK
0.00 43.43 us 28.00 us 214.00 us 93 STATFS
0.00 246.35 us 11.00 us 1155.00 us 20 READDIR
0.00 283.00 us 233.00 us 353.00 us 18 READDIRP
0.00 153.23 us 122.00 us 259.00 us 87 MKNOD
0.01 99.77 us 10.00 us 258.00 us 442 LOOKUP
0.31 49.22 us 27.00 us 540.00 us 45620 FXATTROP
0.77 124.24 us 87.00 us 604.00 us 44968 WRITE
0.93 15767.71 us 15.00 us 305833.00 us 431 ENTRYLK
1.99 160711.39 us 3332.00 us 406037.00 us 90 UNLINK
96.00 5167.82 us 18.00 us 55972.00 us 135349 FINODELK
Duration: 380 seconds
Data Read: 1048576 bytes
Data Written: 5894045696 bytes
Interval 0 Stats:
Block Size: 131072b+
No. of Reads: 8
No. of Writes: 44968
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 3 FORGET
0.00 0.00 us 0.00 us 0.00 us 35 RELEASE
0.00 0.00 us 0.00 us 0.00 us 28 RELEASEDIR
0.00 78.00 us 78.00 us 78.00 us 1 FSTAT
0.00 35.67 us 26.00 us 73.00 us 6 FLUSH
0.00 324.00 us 324.00 us 324.00 us 1 XATTROP
0.00 45.80 us 38.00 us 54.00 us 10 STAT
0.00 227.67 us 216.00 us 242.00 us 3 CREATE
0.00 113.38 us 68.00 us 381.00 us 8 READ
0.00 39.82 us 1.00 us 148.00 us 28 OPENDIR
0.00 67.54 us 10.00 us 283.00 us 24 GETXATTR
0.00 59.97 us 45.00 us 113.00 us 32 OPEN
0.00 24.41 us 13.00 us 89.00 us 161 INODELK
0.00 43.43 us 28.00 us 214.00 us 93 STATFS
0.00 246.35 us 11.00 us 1155.00 us 20 READDIR
0.00 283.00 us 233.00 us 353.00 us 18 READDIRP
0.00 153.23 us 122.00 us 259.00 us 87 MKNOD
0.01 99.77 us 10.00 us 258.00 us 442 LOOKUP
0.31 49.22 us 27.00 us 540.00 us 45620 FXATTROP
0.77 124.24 us 87.00 us 604.00 us 44968 WRITE
0.93 15767.71 us 15.00 us 305833.00 us 431 ENTRYLK
1.99 160711.39 us 3332.00 us 406037.00 us 90 UNLINK
96.00 5167.82 us 18.00 us 55972.00 us 135349 FINODELK
Duration: 380 seconds
Data Read: 1048576 bytes
Data Written: 5894045696 bytes
Brick: ovirt3.localdomain:/gluster_bricks/data/data
---------------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 39328
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2 FORGET
0.00 0.00 us 0.00 us 0.00 us 12 RELEASE
0.00 0.00 us 0.00 us 0.00 us 17 RELEASEDIR
0.00 101.00 us 101.00 us 101.00 us 1 FSTAT
0.00 51.50 us 20.00 us 81.00 us 4 FLUSH
0.01 219.50 us 188.00 us 251.00 us 2 CREATE
0.01 43.45 us 11.00 us 90.00 us 11 GETXATTR
0.01 62.30 us 38.00 us 119.00 us 10 OPEN
0.01 50.59 us 1.00 us 102.00 us 17 OPENDIR
0.01 24.60 us 12.00 us 64.00 us 40 INODELK
0.02 176.30 us 10.00 us 765.00 us 10 READDIR
0.07 63.08 us 39.00 us 133.00 us 78 UNLINK
0.13 27.35 us 10.00 us 91.00 us 333 ENTRYLK
0.13 126.89 us 99.00 us 179.00 us 76 MKNOD
0.42 116.70 us 8.00 us 8661.00 us 261 LOOKUP
28.73 51.79 us 22.00 us 2574.00 us 39822 FXATTROP
29.52 53.87 us 16.00 us 3290.00 us 39328 WRITE
40.92 24.71 us 10.00 us 3224.00 us 118864 FINODELK
Duration: 189 seconds
Data Read: 0 bytes
Data Written: 39328 bytes
Interval 0 Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 39328
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2 FORGET
0.00 0.00 us 0.00 us 0.00 us 12 RELEASE
0.00 0.00 us 0.00 us 0.00 us 17 RELEASEDIR
0.00 101.00 us 101.00 us 101.00 us 1 FSTAT
0.00 51.50 us 20.00 us 81.00 us 4 FLUSH
0.01 219.50 us 188.00 us 251.00 us 2 CREATE
0.01 43.45 us 11.00 us 90.00 us 11 GETXATTR
0.01 62.30 us 38.00 us 119.00 us 10 OPEN
0.01 50.59 us 1.00 us 102.00 us 17 OPENDIR
0.01 24.60 us 12.00 us 64.00 us 40 INODELK
0.02 176.30 us 10.00 us 765.00 us 10 READDIR
0.07 63.08 us 39.00 us 133.00 us 78 UNLINK
0.13 27.35 us 10.00 us 91.00 us 333 ENTRYLK
0.13 126.89 us 99.00 us 179.00 us 76 MKNOD
0.42 116.70 us 8.00 us 8661.00 us 261 LOOKUP
28.73 51.79 us 22.00 us 2574.00 us 39822 FXATTROP
29.52 53.87 us 16.00 us 3290.00 us 39328 WRITE
40.92 24.71 us 10.00 us 3224.00 us 118864 FINODELK
Duration: 189 seconds
Data Read: 0 bytes
Data Written: 39328 bytes
Brick: ovirt2.localdomain:/gluster_bricks/data/data
---------------------------------------------------
Cumulative Stats:
Block Size: 512b+ 131072b+
No. of Reads: 0 0
No. of Writes: 36 76758
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6 FORGET
0.00 0.00 us 0.00 us 0.00 us 87 RELEASE
0.00 0.00 us 0.00 us 0.00 us 96 RELEASEDIR
0.00 100.50 us 80.00 us 121.00 us 2 REMOVEXATTR
0.00 101.00 us 101.00 us 101.00 us 2 SETXATTR
0.00 36.18 us 22.00 us 62.00 us 11 FLUSH
0.00 57.44 us 42.00 us 77.00 us 9 FTRUNCATE
0.00 82.56 us 59.00 us 138.00 us 9 FSTAT
0.00 89.42 us 67.00 us 161.00 us 12 SETATTR
0.00 272.40 us 235.00 us 296.00 us 5 CREATE
0.01 154.28 us 88.00 us 320.00 us 18 XATTROP
0.01 45.29 us 1.00 us 319.00 us 96 OPENDIR
0.01 86.69 us 30.00 us 379.00 us 62 STAT
0.01 64.30 us 47.00 us 169.00 us 84 OPEN
0.02 107.34 us 23.00 us 273.00 us 73 READDIRP
0.02 4688.00 us 86.00 us 9290.00 us 2 TRUNCATE
0.02 59.29 us 13.00 us 394.00 us 165 GETXATTR
0.03 128.51 us 27.00 us 338.00 us 96 FSYNC
0.03 240.75 us 14.00 us 1943.00 us 52 READDIR
0.04 65.59 us 26.00 us 293.00 us 279 STATFS
0.06 180.77 us 118.00 us 306.00 us 148 MKNOD
0.14 37.98 us 17.00 us 192.00 us 1598 INODELK
0.67 91.68 us 12.00 us 1141.00 us 3186 LOOKUP
10.10 55.92 us 28.00 us 1658.00 us 78608 FXATTROP
11.89 6814.76 us 18.00 us 301246.00 us 760 ENTRYLK
19.44 36.55 us 14.00 us 2353.00 us 231535 FINODELK
25.21 142.92 us 62.00 us 593.00 us 76794 WRITE
32.28 91283.68 us 28.00 us 316658.00 us 154 UNLINK
Duration: 1206 seconds
Data Read: 0 bytes
Data Written: 10060843008 bytes
Interval 0 Stats:
Block Size: 512b+ 131072b+
No. of Reads: 0 0
No. of Writes: 36 76758
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6 FORGET
0.00 0.00 us 0.00 us 0.00 us 87 RELEASE
0.00 0.00 us 0.00 us 0.00 us 96 RELEASEDIR
0.00 100.50 us 80.00 us 121.00 us 2 REMOVEXATTR
0.00 101.00 us 101.00 us 101.00 us 2 SETXATTR
0.00 36.18 us 22.00 us 62.00 us 11 FLUSH
0.00 57.44 us 42.00 us 77.00 us 9 FTRUNCATE
0.00 82.56 us 59.00 us 138.00 us 9 FSTAT
0.00 89.42 us 67.00 us 161.00 us 12 SETATTR
0.00 272.40 us 235.00 us 296.00 us 5 CREATE
0.01 154.28 us 88.00 us 320.00 us 18 XATTROP
0.01 45.29 us 1.00 us 319.00 us 96 OPENDIR
0.01 86.69 us 30.00 us 379.00 us 62 STAT
0.01 64.30 us 47.00 us 169.00 us 84 OPEN
0.02 107.34 us 23.00 us 273.00 us 73 READDIRP
0.02 4688.00 us 86.00 us 9290.00 us 2 TRUNCATE
0.02 59.29 us 13.00 us 394.00 us 165 GETXATTR
0.03 128.51 us 27.00 us 338.00 us 96 FSYNC
0.03 240.75 us 14.00 us 1943.00 us 52 READDIR
0.04 65.59 us 26.00 us 293.00 us 279 STATFS
0.06 180.77 us 118.00 us 306.00 us 148 MKNOD
0.14 37.98 us 17.00 us 192.00 us 1598 INODELK
0.67 91.66 us 12.00 us 1141.00 us 3186 LOOKUP
10.10 55.92 us 28.00 us 1658.00 us 78608 FXATTROP
11.89 6814.76 us 18.00 us 301246.00 us 760 ENTRYLK
19.44 36.55 us 14.00 us 2353.00 us 231535 FINODELK
25.21 142.92 us 62.00 us 593.00 us 76794 WRITE
32.28 91283.68 us 28.00 us 316658.00 us 154 UNLINK
Duration: 1206 seconds
Data Read: 0 bytes
Data Written: 10060843008 bytes
This indicates to me that it's not a problem in Disk/LVM/FileSystem layout.
Most probably I haven't created the volume properly or some option/feature is disabled ?!?
Network shows OK for a gigabit:
[root at ovirt1 data]# dd if=/dev/zero status=progress | nc ovirt2 9999
3569227264 bytes (3.6 GB) copied, 29.001052 s, 123 MB/s^C
7180980+0 records in
7180979+0 records out
3676661248 bytes (3.7 GB) copied, 29.8739 s, 123 MB/s
I'm looking for any help... you can share your volume info also.
Thanks in advance.
Best Regards,
Strahil Nikolov
More information about the Gluster-users
mailing list