[Gluster-users] Performance issue, need guidance
Strahil Nikolov
hunter86_bg at yahoo.com
Tue Jan 22 23:09:00 UTC 2019
Hello Community,
I would be very grateful if you share your thoughts about my problem.I'm quite new to gluster , so keep that in mind.
Setup:I have 3 nodes for an ovirt setup (2 hosts + 1 gluster arbiter). The diagram of my setup is: ovirt
|
|
|
| | |
|
|
|
| |
ovirt
| |
|
|
I'm having bad write & read performance from the VM, despite having a SSD (not an enterprise-grade one , but still a SSD) for LVM writeback cache. Writing directly to the bricks is far faster than my network can do (1 Gbit/s network). I have checked the bandwidth and it seems that maximum possible for me is 123MB/s, yet observed speeds via glusterfs fuse client (testing from one of the hosts) is no more than 56MB/s and from VMs is around 20-30 MB/s.
Here is my volume profile info (I'm not sure what to look for):
[root at ovirt2 tuned]# gluster volume profile data infoBrick: ovirt2.localdomain:/gluster_bricks/data/data---------------------------------------------------Cumulative Stats: Block Size: 256b+ 512b+ 1024b+ No. of Reads: 5854 198 159 No. of Writes: 2 6025 1430 Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 302 9950 6485 No. of Writes: 611 23513 6540 Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 6952 1774 1699 No. of Writes: 6439 5870 5171 Block Size: 131072b+ No. of Reads: 48690 No. of Writes: 127023 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 14 FORGET 0.00 0.00 us 0.00 us 0.00 us 534 RELEASE 0.00 0.00 us 0.00 us 0.00 us 2180 RELEASEDIR 0.00 44.00 us 44.00 us 44.00 us 1 GETXATTR 0.00 28.75 us 23.00 us 33.00 us 4 INODELK 0.00 85.00 us 85.00 us 85.00 us 2 SETATTR 0.00 52.50 us 31.00 us 113.00 us 8 OPENDIR 0.00 33.83 us 23.00 us 60.00 us 29 ENTRYLK 0.00 40.32 us 24.00 us 157.00 us 53 STAT 0.00 59.32 us 43.00 us 121.00 us 38 FSTAT 0.00 188.08 us 28.00 us 458.00 us 13 READDIRP 0.00 323.46 us 286.00 us 398.00 us 13 MKNOD 0.00 42.84 us 26.00 us 142.00 us 109 STATFS 0.01 147.36 us 71.00 us 365.00 us 200 LOOKUP 0.17 914.71 us 186.00 us 3887.00 us 533 READ 3.57 322.84 us 41.00 us 1552044.00 us 32308 FXATTROP 21.39 1090.04 us 15.00 us 1228946.00 us 57261 FINODELK 27.42 2527.96 us 98.00 us 1552471.00 us 31651 WRITE 47.43 12047.80 us 203.00 us 1891369.00 us 11489 FSYNC Duration: 59012 seconds Data Read: 6853540304 bytesData Written: 17667709568 bytes Interval 0 Stats: Block Size: 256b+ 512b+ 1024b+ No. of Reads: 5854 198 159 No. of Writes: 2 6025 1430 Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 302 9950 6485 No. of Writes: 611 23513 6540 Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 6952 1774 1699 No. of Writes: 6439 5870 5171 Block Size: 131072b+ No. of Reads: 48690 No. of Writes: 127023 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 14 FORGET 0.00 0.00 us 0.00 us 0.00 us 534 RELEASE 0.00 0.00 us 0.00 us 0.00 us 2180 RELEASEDIR 0.00 44.00 us 44.00 us 44.00 us 1 GETXATTR 0.00 28.75 us 23.00 us 33.00 us 4 INODELK 0.00 85.00 us 85.00 us 85.00 us 2 SETATTR 0.00 52.50 us 31.00 us 113.00 us 8 OPENDIR 0.00 33.83 us 23.00 us 60.00 us 29 ENTRYLK 0.00 40.32 us 24.00 us 157.00 us 53 STAT 0.00 59.32 us 43.00 us 121.00 us 38 FSTAT 0.00 188.08 us 28.00 us 458.00 us 13 READDIRP 0.00 323.46 us 286.00 us 398.00 us 13 MKNOD 0.00 42.84 us 26.00 us 142.00 us 109 STATFS 0.01 147.36 us 71.00 us 365.00 us 200 LOOKUP 0.17 914.71 us 186.00 us 3887.00 us 533 READ 3.57 322.84 us 41.00 us 1552044.00 us 32308 FXATTROP 21.39 1090.04 us 15.00 us 1228946.00 us 57261 FINODELK 27.42 2527.96 us 98.00 us 1552471.00 us 31651 WRITE 47.43 12047.80 us 203.00 us 1891369.00 us 11489 FSYNC Duration: 59012 seconds Data Read: 6853540304 bytesData Written: 17667709568 bytes Brick: ovirt3.localdomain:/gluster_bricks/data/data---------------------------------------------------Cumulative Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 257997 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 53 FORGET 0.00 0.00 us 0.00 us 0.00 us 3180 RELEASE 0.00 0.00 us 0.00 us 0.00 us 2672 RELEASEDIR 0.00 14.33 us 13.00 us 16.00 us 3 FLUSH 0.00 43.00 us 43.00 us 43.00 us 1 TRUNCATE 0.00 229.00 us 229.00 us 229.00 us 1 CREATE 0.00 80.00 us 48.00 us 114.00 us 3 XATTROP 0.00 66.83 us 57.00 us 79.00 us 6 SETATTR 0.00 29.76 us 1.00 us 72.00 us 34 OPENDIR 0.00 96.12 us 12.00 us 240.00 us 17 GETXATTR 0.01 375.00 us 216.00 us 790.00 us 6 READDIR 0.01 30.90 us 11.00 us 186.00 us 73 INODELK 0.01 178.89 us 36.00 us 541.00 us 19 OPEN 0.02 113.86 us 57.00 us 313.00 us 78 UNLINK 0.05 212.36 us 146.00 us 488.00 us 90 MKNOD 0.14 35.88 us 10.00 us 399.00 us 1636 ENTRYLK 0.42 80.78 us 21.00 us 495.00 us 2122 LOOKUP 7.92 44.74 us 12.00 us 10903.00 us 71878 WRITE 11.52 62.55 us 25.00 us 27390.00 us 74768 FXATTROP 12.00 27.54 us 9.00 us 7191.00 us 176968 FINODELK 67.90 2384.96 us 53.00 us 82033.00 us 11562 FSYNC Duration: 75025 seconds Data Read: 0 bytesData Written: 257997 bytes Interval 2 Stats: Block Size: 1b+ No. of Reads: 0 No. of Writes: 201556 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 19 FORGET 0.00 0.00 us 0.00 us 0.00 us 2046 RELEASE 0.00 0.00 us 0.00 us 0.00 us 2606 RELEASEDIR 0.00 38.50 us 35.00 us 42.00 us 2 GETXATTR 0.00 33.17 us 15.00 us 62.00 us 6 INODELK 0.00 71.00 us 63.00 us 79.00 us 3 SETATTR 0.00 41.75 us 24.00 us 72.00 us 12 OPENDIR 0.00 26.28 us 12.00 us 48.00 us 29 ENTRYLK 0.01 191.15 us 154.00 us 240.00 us 13 MKNOD 0.10 112.97 us 40.00 us 170.00 us 299 LOOKUP 4.22 45.65 us 14.00 us 1288.00 us 31732 WRITE 5.99 35.51 us 9.00 us 7124.00 us 57863 FINODELK 9.57 101.29 us 33.00 us 27390.00 us 32435 FXATTROP 80.11 2390.06 us 61.00 us 82033.00 us 11507 FSYNC Duration: 74302 seconds Data Read: 0 bytesData Written: 201556 bytes Brick: ovirt1.localdomain:/gluster_bricks/data/data---------------------------------------------------Cumulative Stats: Block Size: 256b+ 512b+ 1024b+ No. of Reads: 13408 183 2200 No. of Writes: 11 12677 1556 Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 60 3393 1278 No. of Writes: 666 59118 22688 Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 10658 574 954 No. of Writes: 32082 29900 61801 Block Size: 131072b+ No. of Reads: 55974 No. of Writes: 576153 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 58 FORGET 0.00 0.00 us 0.00 us 0.00 us 3357 RELEASE 0.00 0.00 us 0.00 us 0.00 us 4823 RELEASEDIR 0.00 26.00 us 24.00 us 28.00 us 3 FLUSH 0.00 64.00 us 52.00 us 81.00 us 3 FSTAT 0.00 202.00 us 202.00 us 202.00 us 1 TRUNCATE 0.00 271.00 us 271.00 us 271.00 us 1 CREATE 0.00 135.33 us 72.00 us 180.00 us 3 XATTROP 0.00 103.50 us 84.00 us 155.00 us 6 SETATTR 0.00 69.77 us 17.00 us 174.00 us 13 GETXATTR 0.00 438.67 us 399.00 us 479.00 us 3 READDIRP 0.00 45.12 us 3.00 us 94.00 us 34 OPENDIR 0.00 91.63 us 53.00 us 184.00 us 19 OPEN 0.00 331.57 us 262.00 us 381.00 us 7 READDIR 0.00 46.28 us 24.00 us 176.00 us 337 STATFS 0.00 223.68 us 101.00 us 630.00 us 78 UNLINK 0.00 318.33 us 176.00 us 456.00 us 90 MKNOD 0.00 237.58 us 18.00 us 148201.00 us 2122 LOOKUP 0.01 1323.15 us 104.00 us 73159.00 us 704 READ 0.02 1890.07 us 15.00 us 89692.00 us 1634 ENTRYLK 0.07 137914.67 us 22.00 us 1523239.00 us 73 INODELK 0.22 395.07 us 30.00 us 676921.00 us 74768 FXATTROP 1.16 13685.56 us 1739.00 us 650863.00 us 11562 FSYNC 3.08 5851.12 us 94.00 us 1033754.00 us 71879 WRITE 95.43 76480.58 us 16.00 us 5368628.00 us 170274 FINODELK Duration: 134846 seconds Data Read: 7665929388 bytesData Written: 84724663200 bytes Interval 2 Stats: Block Size: 256b+ 512b+ 1024b+ No. of Reads: 7432 130 1738 No. of Writes: 6 6816 1430 Block Size: 2048b+ 4096b+ 8192b+ No. of Reads: 48 3099 1040 No. of Writes: 611 24732 6649 Block Size: 16384b+ 32768b+ 65536b+ No. of Reads: 9382 442 832 No. of Writes: 6602 6234 5531 Block Size: 131072b+ No. of Reads: 20719 No. of Writes: 142946 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop --------- ----------- ----------- ----------- ------------ ---- 0.00 0.00 us 0.00 us 0.00 us 19 FORGET 0.00 0.00 us 0.00 us 0.00 us 2046 RELEASE 0.00 0.00 us 0.00 us 0.00 us 2606 RELEASEDIR 0.00 63.00 us 47.00 us 79.00 us 2 GETXATTR 0.00 64.00 us 52.00 us 81.00 us 3 FSTAT 0.00 33.83 us 27.00 us 41.00 us 6 INODELK 0.00 116.33 us 92.00 us 155.00 us 3 SETATTR 0.00 479.00 us 479.00 us 479.00 us 1 READDIRP 0.00 48.75 us 32.00 us 83.00 us 12 OPENDIR 0.00 283.31 us 176.00 us 383.00 us 13 MKNOD 0.00 46.40 us 26.00 us 142.00 us 164 STATFS 0.00 1303.45 us 20.00 us 36916.00 us 29 ENTRYLK 0.00 1054.11 us 63.00 us 148201.00 us 299 LOOKUP 0.01 1302.85 us 104.00 us 73159.00 us 618 READ 0.20 812.06 us 40.00 us 676921.00 us 32435 FXATTROP 1.19 13697.54 us 1739.00 us 650863.00 us 11507 FSYNC 1.94 8117.26 us 98.00 us 1033754.00 us 31733 WRITE 96.67 229052.02 us 17.00 us 5368628.00 us 56103 FINODELK Duration: 74302 seconds Data Read: 2999932124 bytesData Written: 19810606944 bytes
Any help will be appreciated.
Best Regards,Strahil Nikolov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190122/9df01481/attachment.html>
More information about the Gluster-users
mailing list