<div dir="ltr">I didn't understand the issue properly. Mostly I missed something.<div><br></div><div>Are you concerned the performance is 49MB/s with and without perf options? or are you expecting it to be 123MB/s as over the n/w you get that speed?</div><div><br></div><div>If it is the first problem, then you are actually having 'performance.write-behind on' in both options, and it is the only perf xlator which comes into action during the test you ran.</div><div><br></div><div>If it is the second, then please be informed that gluster does client side replication, which means, n/w would be split in half for write operations (like write(), creat() etc), so the number you are getting is almost the maximum with 1GbE.</div><div><br></div><div>Regards,</div><div>Amar</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 23, 2019 at 8:38 PM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello Community,<br>
<br>
recently I have built a new lab based on oVirt and CentOS 7.<br>
During deployment I had some hicups, but now the engine is up and running - but gluster is causing me trouble.<br>
<br>
Symptoms: Slow VM install from DVD, poor write performance. The latter has been tested via:<br>
dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_data bs=1M count=1000 status=progress<br>
<br>
The reported speed is 60MB/s which is way too low for my setup.<br>
<br>
My lab design:<br>
<a href="https://drive.google.com/file/d/1SiW21ASPXHRAEuE_jZ50R3FoO-NcnFqT/view?usp=sharing" rel="noreferrer" target="_blank">https://drive.google.com/file/d/1SiW21ASPXHRAEuE_jZ50R3FoO-NcnFqT/view?usp=sharing</a><br>
Gluster version is 3.12.15<br>
<br>
So far I have done:<br>
<br>
1. Added 'server.allow-insecure on' (with 'option rpc-auth-allow-insecure on' in glusterd.vol)<br>
Volume info after that change:<br>
<br>
Volume Name: data<br>
Type: Replicate<br>
Volume ID: 9b06a1e9-8102-4cd7-bc56-84960a1efaa2<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 1 x (2 + 1) = 3<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: ovirt1.localdomain:/gluster_bricks/data/data<br>
Brick2: ovirt2.localdomain:/gluster_bricks/data/data<br>
Brick3: ovirt3.localdomain:/gluster_bricks/data/data (arbiter)<br>
Options Reconfigured:<br>
performance.client-io-threads: off<br>
nfs.disable: on<br>
transport.address-family: inet<br>
performance.quick-read: off<br>
performance.read-ahead: off<br>
performance.io-cache: off<br>
performance.low-prio-threads: 32<br>
network.remote-dio: off<br>
cluster.eager-lock: enable<br>
cluster.quorum-type: auto<br>
cluster.server-quorum-type: server<br>
cluster.data-self-heal-algorithm: full<br>
cluster.locking-scheme: granular<br>
cluster.shd-max-threads: 8<br>
cluster.shd-wait-qlength: 10000<br>
features.shard: on<br>
user.cifs: off<br>
storage.owner-uid: 36<br>
storage.owner-gid: 36<br>
network.ping-timeout: 30<br>
performance.strict-o-direct: on<br>
cluster.granular-entry-heal: enable<br>
server.allow-insecure: on<br>
<br>
Seems no positive or negative effect so far.<br>
<br>
2. Tested with tmpfs on all bricks -> ovirt1 mounted gluster volume -> max 60MB/s (bs=1M without 'oflag=direct')<br>
<br>
<br>
[root@ovirt1 data]# dd if=/dev/zero of=large_io bs=1M count=4000 status=progress<br>
4177526784 bytes (4.2 GB) copied, 70.843409 s, 59.0 MB/s<br>
4000+0 records in<br>
4000+0 records out<br>
4194304000 bytes (4.2 GB) copied, 71.1407 s, 59.0 MB/s<br>
[root@ovirt1 data]# rm -f large_io<br>
[root@ovirt1 data]# gluster volume profile data info<br>
Brick: ovirt1.localdomain:/gluster_bricks/data/data<br>
---------------------------------------------------<br>
Cumulative Stats:<br>
Block Size: 131072b+ <br>
No. of Reads: 8 <br>
No. of Writes: 44968 <br>
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br>
--------- ----------- ----------- ----------- ------------ ----<br>
0.00 0.00 us 0.00 us 0.00 us 3 FORGET<br>
0.00 0.00 us 0.00 us 0.00 us 35 RELEASE<br>
0.00 0.00 us 0.00 us 0.00 us 28 RELEASEDIR<br>
0.00 78.00 us 78.00 us 78.00 us 1 FSTAT<br>
0.00 35.67 us 26.00 us 73.00 us 6 FLUSH<br>
0.00 324.00 us 324.00 us 324.00 us 1 XATTROP<br>
0.00 45.80 us 38.00 us 54.00 us 10 STAT<br>
0.00 227.67 us 216.00 us 242.00 us 3 CREATE<br>
0.00 113.38 us 68.00 us 381.00 us 8 READ<br>
0.00 39.82 us 1.00 us 148.00 us 28 OPENDIR<br>
0.00 67.54 us 10.00 us 283.00 us 24 GETXATTR<br>
0.00 59.97 us 45.00 us 113.00 us 32 OPEN<br>
0.00 24.41 us 13.00 us 89.00 us 161 INODELK<br>
0.00 43.43 us 28.00 us 214.00 us 93 STATFS<br>
0.00 246.35 us 11.00 us 1155.00 us 20 READDIR<br>
0.00 283.00 us 233.00 us 353.00 us 18 READDIRP<br>
0.00 153.23 us 122.00 us 259.00 us 87 MKNOD<br>
0.01 99.77 us 10.00 us 258.00 us 442 LOOKUP<br>
0.31 49.22 us 27.00 us 540.00 us 45620 FXATTROP<br>
0.77 124.24 us 87.00 us 604.00 us 44968 WRITE<br>
0.93 15767.71 us 15.00 us 305833.00 us 431 ENTRYLK<br>
1.99 160711.39 us 3332.00 us 406037.00 us 90 UNLINK<br>
96.00 5167.82 us 18.00 us 55972.00 us 135349 FINODELK<br>
<br>
Duration: 380 seconds<br>
Data Read: 1048576 bytes<br>
Data Written: 5894045696 bytes<br>
<br>
Interval 0 Stats:<br>
Block Size: 131072b+ <br>
No. of Reads: 8 <br>
No. of Writes: 44968 <br>
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br>
--------- ----------- ----------- ----------- ------------ ----<br>
0.00 0.00 us 0.00 us 0.00 us 3 FORGET<br>
0.00 0.00 us 0.00 us 0.00 us 35 RELEASE<br>
0.00 0.00 us 0.00 us 0.00 us 28 RELEASEDIR<br>
0.00 78.00 us 78.00 us 78.00 us 1 FSTAT<br>
0.00 35.67 us 26.00 us 73.00 us 6 FLUSH<br>
0.00 324.00 us 324.00 us 324.00 us 1 XATTROP<br>
0.00 45.80 us 38.00 us 54.00 us 10 STAT<br>
0.00 227.67 us 216.00 us 242.00 us 3 CREATE<br>
0.00 113.38 us 68.00 us 381.00 us 8 READ<br>
0.00 39.82 us 1.00 us 148.00 us 28 OPENDIR<br>
0.00 67.54 us 10.00 us 283.00 us 24 GETXATTR<br>
0.00 59.97 us 45.00 us 113.00 us 32 OPEN<br>
0.00 24.41 us 13.00 us 89.00 us 161 INODELK<br>
0.00 43.43 us 28.00 us 214.00 us 93 STATFS<br>
0.00 246.35 us 11.00 us 1155.00 us 20 READDIR<br>
0.00 283.00 us 233.00 us 353.00 us 18 READDIRP<br>
0.00 153.23 us 122.00 us 259.00 us 87 MKNOD<br>
0.01 99.77 us 10.00 us 258.00 us 442 LOOKUP<br>
0.31 49.22 us 27.00 us 540.00 us 45620 FXATTROP<br>
0.77 124.24 us 87.00 us 604.00 us 44968 WRITE<br>
0.93 15767.71 us 15.00 us 305833.00 us 431 ENTRYLK<br>
1.99 160711.39 us 3332.00 us 406037.00 us 90 UNLINK<br>
96.00 5167.82 us 18.00 us 55972.00 us 135349 FINODELK<br>
<br>
Duration: 380 seconds<br>
Data Read: 1048576 bytes<br>
Data Written: 5894045696 bytes<br>
<br>
Brick: ovirt3.localdomain:/gluster_bricks/data/data<br>
---------------------------------------------------<br>
Cumulative Stats:<br>
Block Size: 1b+ <br>
No. of Reads: 0 <br>
No. of Writes: 39328 <br>
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br>
--------- ----------- ----------- ----------- ------------ ----<br>
0.00 0.00 us 0.00 us 0.00 us 2 FORGET<br>
0.00 0.00 us 0.00 us 0.00 us 12 RELEASE<br>
0.00 0.00 us 0.00 us 0.00 us 17 RELEASEDIR<br>
0.00 101.00 us 101.00 us 101.00 us 1 FSTAT<br>
0.00 51.50 us 20.00 us 81.00 us 4 FLUSH<br>
0.01 219.50 us 188.00 us 251.00 us 2 CREATE<br>
0.01 43.45 us 11.00 us 90.00 us 11 GETXATTR<br>
0.01 62.30 us 38.00 us 119.00 us 10 OPEN<br>
0.01 50.59 us 1.00 us 102.00 us 17 OPENDIR<br>
0.01 24.60 us 12.00 us 64.00 us 40 INODELK<br>
0.02 176.30 us 10.00 us 765.00 us 10 READDIR<br>
0.07 63.08 us 39.00 us 133.00 us 78 UNLINK<br>
0.13 27.35 us 10.00 us 91.00 us 333 ENTRYLK<br>
0.13 126.89 us 99.00 us 179.00 us 76 MKNOD<br>
0.42 116.70 us 8.00 us 8661.00 us 261 LOOKUP<br>
28.73 51.79 us 22.00 us 2574.00 us 39822 FXATTROP<br>
29.52 53.87 us 16.00 us 3290.00 us 39328 WRITE<br>
40.92 24.71 us 10.00 us 3224.00 us 118864 FINODELK<br>
<br>
Duration: 189 seconds<br>
Data Read: 0 bytes<br>
Data Written: 39328 bytes<br>
<br>
Interval 0 Stats:<br>
Block Size: 1b+ <br>
No. of Reads: 0 <br>
No. of Writes: 39328 <br>
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br>
--------- ----------- ----------- ----------- ------------ ----<br>
0.00 0.00 us 0.00 us 0.00 us 2 FORGET<br>
0.00 0.00 us 0.00 us 0.00 us 12 RELEASE<br>
0.00 0.00 us 0.00 us 0.00 us 17 RELEASEDIR<br>
0.00 101.00 us 101.00 us 101.00 us 1 FSTAT<br>
0.00 51.50 us 20.00 us 81.00 us 4 FLUSH<br>
0.01 219.50 us 188.00 us 251.00 us 2 CREATE<br>
0.01 43.45 us 11.00 us 90.00 us 11 GETXATTR<br>
0.01 62.30 us 38.00 us 119.00 us 10 OPEN<br>
0.01 50.59 us 1.00 us 102.00 us 17 OPENDIR<br>
0.01 24.60 us 12.00 us 64.00 us 40 INODELK<br>
0.02 176.30 us 10.00 us 765.00 us 10 READDIR<br>
0.07 63.08 us 39.00 us 133.00 us 78 UNLINK<br>
0.13 27.35 us 10.00 us 91.00 us 333 ENTRYLK<br>
0.13 126.89 us 99.00 us 179.00 us 76 MKNOD<br>
0.42 116.70 us 8.00 us 8661.00 us 261 LOOKUP<br>
28.73 51.79 us 22.00 us 2574.00 us 39822 FXATTROP<br>
29.52 53.87 us 16.00 us 3290.00 us 39328 WRITE<br>
40.92 24.71 us 10.00 us 3224.00 us 118864 FINODELK<br>
<br>
Duration: 189 seconds<br>
Data Read: 0 bytes<br>
Data Written: 39328 bytes<br>
<br>
Brick: ovirt2.localdomain:/gluster_bricks/data/data<br>
---------------------------------------------------<br>
Cumulative Stats:<br>
Block Size: 512b+ 131072b+ <br>
No. of Reads: 0 0 <br>
No. of Writes: 36 76758 <br>
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br>
--------- ----------- ----------- ----------- ------------ ----<br>
0.00 0.00 us 0.00 us 0.00 us 6 FORGET<br>
0.00 0.00 us 0.00 us 0.00 us 87 RELEASE<br>
0.00 0.00 us 0.00 us 0.00 us 96 RELEASEDIR<br>
0.00 100.50 us 80.00 us 121.00 us 2 REMOVEXATTR<br>
0.00 101.00 us 101.00 us 101.00 us 2 SETXATTR<br>
0.00 36.18 us 22.00 us 62.00 us 11 FLUSH<br>
0.00 57.44 us 42.00 us 77.00 us 9 FTRUNCATE<br>
0.00 82.56 us 59.00 us 138.00 us 9 FSTAT<br>
0.00 89.42 us 67.00 us 161.00 us 12 SETATTR<br>
0.00 272.40 us 235.00 us 296.00 us 5 CREATE<br>
0.01 154.28 us 88.00 us 320.00 us 18 XATTROP<br>
0.01 45.29 us 1.00 us 319.00 us 96 OPENDIR<br>
0.01 86.69 us 30.00 us 379.00 us 62 STAT<br>
0.01 64.30 us 47.00 us 169.00 us 84 OPEN<br>
0.02 107.34 us 23.00 us 273.00 us 73 READDIRP<br>
0.02 4688.00 us 86.00 us 9290.00 us 2 TRUNCATE<br>
0.02 59.29 us 13.00 us 394.00 us 165 GETXATTR<br>
0.03 128.51 us 27.00 us 338.00 us 96 FSYNC<br>
0.03 240.75 us 14.00 us 1943.00 us 52 READDIR<br>
0.04 65.59 us 26.00 us 293.00 us 279 STATFS<br>
0.06 180.77 us 118.00 us 306.00 us 148 MKNOD<br>
0.14 37.98 us 17.00 us 192.00 us 1598 INODELK<br>
0.67 91.68 us 12.00 us 1141.00 us 3186 LOOKUP<br>
10.10 55.92 us 28.00 us 1658.00 us 78608 FXATTROP<br>
11.89 6814.76 us 18.00 us 301246.00 us 760 ENTRYLK<br>
19.44 36.55 us 14.00 us 2353.00 us 231535 FINODELK<br>
25.21 142.92 us 62.00 us 593.00 us 76794 WRITE<br>
32.28 91283.68 us 28.00 us 316658.00 us 154 UNLINK<br>
<br>
Duration: 1206 seconds<br>
Data Read: 0 bytes<br>
Data Written: 10060843008 bytes<br>
<br>
Interval 0 Stats:<br>
Block Size: 512b+ 131072b+ <br>
No. of Reads: 0 0 <br>
No. of Writes: 36 76758 <br>
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br>
--------- ----------- ----------- ----------- ------------ ----<br>
0.00 0.00 us 0.00 us 0.00 us 6 FORGET<br>
0.00 0.00 us 0.00 us 0.00 us 87 RELEASE<br>
0.00 0.00 us 0.00 us 0.00 us 96 RELEASEDIR<br>
0.00 100.50 us 80.00 us 121.00 us 2 REMOVEXATTR<br>
0.00 101.00 us 101.00 us 101.00 us 2 SETXATTR<br>
0.00 36.18 us 22.00 us 62.00 us 11 FLUSH<br>
0.00 57.44 us 42.00 us 77.00 us 9 FTRUNCATE<br>
0.00 82.56 us 59.00 us 138.00 us 9 FSTAT<br>
0.00 89.42 us 67.00 us 161.00 us 12 SETATTR<br>
0.00 272.40 us 235.00 us 296.00 us 5 CREATE<br>
0.01 154.28 us 88.00 us 320.00 us 18 XATTROP<br>
0.01 45.29 us 1.00 us 319.00 us 96 OPENDIR<br>
0.01 86.69 us 30.00 us 379.00 us 62 STAT<br>
0.01 64.30 us 47.00 us 169.00 us 84 OPEN<br>
0.02 107.34 us 23.00 us 273.00 us 73 READDIRP<br>
0.02 4688.00 us 86.00 us 9290.00 us 2 TRUNCATE<br>
0.02 59.29 us 13.00 us 394.00 us 165 GETXATTR<br>
0.03 128.51 us 27.00 us 338.00 us 96 FSYNC<br>
0.03 240.75 us 14.00 us 1943.00 us 52 READDIR<br>
0.04 65.59 us 26.00 us 293.00 us 279 STATFS<br>
0.06 180.77 us 118.00 us 306.00 us 148 MKNOD<br>
0.14 37.98 us 17.00 us 192.00 us 1598 INODELK<br>
0.67 91.66 us 12.00 us 1141.00 us 3186 LOOKUP<br>
10.10 55.92 us 28.00 us 1658.00 us 78608 FXATTROP<br>
11.89 6814.76 us 18.00 us 301246.00 us 760 ENTRYLK<br>
19.44 36.55 us 14.00 us 2353.00 us 231535 FINODELK<br>
25.21 142.92 us 62.00 us 593.00 us 76794 WRITE<br>
32.28 91283.68 us 28.00 us 316658.00 us 154 UNLINK<br>
<br>
Duration: 1206 seconds<br>
Data Read: 0 bytes<br>
Data Written: 10060843008 bytes<br>
<br>
<br>
<br>
This indicates to me that it's not a problem in Disk/LVM/FileSystem layout.<br>
<br>
Most probably I haven't created the volume properly or some option/feature is disabled ?!?<br>
Network shows OK for a gigabit:<br>
[root@ovirt1 data]# dd if=/dev/zero status=progress | nc ovirt2 9999<br>
3569227264 bytes (3.6 GB) copied, 29.001052 s, 123 MB/s^C<br>
7180980+0 records in<br>
7180979+0 records out<br>
3676661248 bytes (3.7 GB) copied, 29.8739 s, 123 MB/s<br>
<br>
<br>
I'm looking for any help... you can share your volume info also.<br>
<br>
Thanks in advance.<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>