<div dir="ltr">Hi,<br><br> Thanks for sharing the output. May be in the case of NVME there is no performance difference in SSL vs Non-SSL.<br> Can you please share the output of the command "gluster v info" and what client you are using to mount the volume?<br> If you are using fuse have u used any specific argument at the time of mounting the volume?<br> Please use default event-threads if you are not getting any performance improvement and share the command output<br> "top -bH -d 10" at the time of writing IO.<br><br>Thanks,<br>Mohit Agrawal<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 6, 2020 at 4:01 AM Michael Richardson <<a href="mailto:hello@mikerichardson.com.au">hello@mikerichardson.com.au</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Mohit and Strahil,<div><br></div><div>Thank you for taking the time to share your advice. Strahil, unfortunately your message didn't come to my inbox, so I'm combining my reply to both yourself and Mohit. </div><div><br></div><div>Mohit, </div><div><br></div><div>I mentioned there was no performance difference between using SSL and not using SSL. I did try setting it to AES128 anyway, but that seemed to have no effect as well. <br><br>Strahil,</div><div><br></div><div>I've had a go at various different settings this morning, including all of those you've supplied. I couldn't see any improvement in throughput when messing with these, but I could make it worse by adjusting things like event-threads. </div><div><br></div><div>I've also tried adjusting the fio test types from one large file to many small files and a mix in between without any real change in peak performance. </div><div><br></div><div>In answer to your other question, I'm running Gluster 7.1-1 on Debian 10. I have confirmed that blk-mq is on and my scheduler is 'none'. I don't seem to have an option to change it to 'noop' but I'm not sure why. </div><div><br></div><div>Here's some profile output as well, in case that's helpful. These' values don't look so bad (to me, at least), but again this is much less than the raw nvme output that I was hoping to get closer to. </div><div><br></div><div><font face="monospace">Brick: node1:/data/nvme/testvol1/brick<br>-------------------------------------------------------------<br>Cumulative Stats:<br> Block Size: 16384b+<br> No. of Reads: 492397<br>No. of Writes: 2849474<br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 57 FORGET<br> 0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE<br> 0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR<br> 0.00 33.67 us 30.56 us 43.31 us 5 GETXATTR<br> 0.01 26.05 us 14.36 us 128.33 us 870 ENTRYLK<br> 0.02 53.48 us 43.23 us 87.31 us 435 FTRUNCATE<br> 0.02 58.54 us 46.75 us 94.94 us 435 FALLOCATE<br> 0.04 24.21 us 9.58 us 679.17 us 2436 FLUSH<br> 0.04 145.95 us 116.32 us 211.72 us 435 CREATE<br> 0.28 211.24 us 32.92 us 17442.92 us 2000 OPEN<br> 0.38 82.16 us 34.00 us 15537.54 us 6929 LOOKUP<br> 1.92 6683.77 us 437.30 us 28616.62 us 436 FSYNC<br> 2.03 48.87 us 7.50 us 24840.37 us 63019 FINODELK<br> 18.97 455.94 us 43.78 us 43701.58 us 63019 FXATTROP<br> 28.06 134.43 us 41.87 us 36537.65 us 316217 WRITE<br> 48.22 611.65 us 100.31 us 43842.80 us 119420 READ<br><br> Duration: 3505 seconds<br> Data Read: 8067432448 bytes<br>Data Written: 46685782016 bytes<br><br>Interval 0 Stats:<br> Block Size: 16384b+<br> No. of Reads: 492397<br>No. of Writes: 2849474<br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 57 FORGET<br> 0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE<br> 0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR<br> 0.00 33.67 us 30.56 us 43.31 us 5 GETXATTR<br> 0.01 26.05 us 14.36 us 128.33 us 870 ENTRYLK<br> 0.02 53.48 us 43.23 us 87.31 us 435 FTRUNCATE<br> 0.02 58.54 us 46.75 us 94.94 us 435 FALLOCATE<br> 0.04 24.21 us 9.58 us 679.17 us 2436 FLUSH<br> 0.04 145.95 us 116.32 us 211.72 us 435 CREATE<br> 0.28 211.24 us 32.92 us 17442.92 us 2000 OPEN<br> 0.38 82.16 us 34.00 us 15537.54 us 6929 LOOKUP<br> 1.92 6683.77 us 437.30 us 28616.62 us 436 FSYNC<br> 2.03 48.87 us 7.50 us 24840.37 us 63019 FINODELK<br> 18.97 455.94 us 43.78 us 43701.58 us 63019 FXATTROP<br> 28.06 134.43 us 41.87 us 36537.65 us 316217 WRITE<br> 48.23 611.65 us 100.31 us 43842.80 us 119420 READ<br><br> Duration: 3505 seconds<br> Data Read: 8067432448 bytes<br>Data Written: 46685782016 bytes<br><br>Brick: node2:/data/nvme/testvol1/brick<br>-------------------------------------------------------------<br>Cumulative Stats:<br> Block Size: 16384b+<br> No. of Reads: 0<br>No. of Writes: 2849474<br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 57 FORGET<br> 0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE<br> 0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR<br> 0.00 27.59 us 26.48 us 28.92 us 5 GETXATTR<br> 0.06 16.31 us 11.92 us 43.37 us 870 ENTRYLK<br> 0.07 40.53 us 36.62 us 163.89 us 435 FTRUNCATE<br> 0.08 43.30 us 38.41 us 70.49 us 435 FALLOCATE<br> 0.13 12.82 us 9.48 us 79.63 us 2436 FLUSH<br> 0.21 114.31 us 96.53 us 166.02 us 435 CREATE<br> 1.53 184.29 us 24.29 us 19723.10 us 2000 OPEN<br> 1.82 63.25 us 25.23 us 19525.54 us 6929 LOOKUP<br> 4.81 18.44 us 9.06 us 182.62 us 63019 FINODELK<br> 11.69 6473.51 us 380.98 us 17129.47 us 436 FSYNC<br> 17.41 66.69 us 44.52 us 12198.65 us 63019 FXATTROP<br> 62.20 47.49 us 31.49 us 16215.18 us 316195 WRITE<br><br> Duration: 3486 seconds<br> Data Read: 0 bytes<br>Data Written: 46685782016 bytes<br><br>Interval 0 Stats:<br> Block Size: 16384b+<br> No. of Reads: 0<br>No. of Writes: 2849474<br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 57 FORGET<br> 0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE<br> 0.00 0.00 us 0.00 us 0.00 us 22 RELEASEDIR<br> 0.00 27.59 us 26.48 us 28.92 us 5 GETXATTR<br> 0.06 16.31 us 11.92 us 43.37 us 870 ENTRYLK<br> 0.07 40.53 us 36.62 us 163.89 us 435 FTRUNCATE<br> 0.08 43.30 us 38.41 us 70.49 us 435 FALLOCATE<br> 0.13 12.82 us 9.48 us 79.63 us 2436 FLUSH<br> 0.21 114.31 us 96.53 us 166.02 us 435 CREATE<br> 1.53 184.29 us 24.29 us 19723.10 us 2000 OPEN<br> 1.82 63.25 us 25.23 us 19525.54 us 6929 LOOKUP<br> 4.81 18.44 us 9.06 us 182.62 us 63019 FINODELK<br> 11.69 6473.51 us 380.98 us 17129.47 us 436 FSYNC<br> 17.41 66.69 us 44.52 us 12198.65 us 63019 FXATTROP<br> 62.20 47.49 us 31.49 us 16215.18 us 316195 WRITE<br><br> Duration: 3486 seconds<br> Data Read: 0 bytes<br>Data Written: 46685782016 bytes<br><br>Brick: node3:/data/nvme/testvol1/brick<br>-------------------------------------------------------------<br>Cumulative Stats:<br> Block Size: 16384b+<br> No. of Reads: 0<br>No. of Writes: 2849474<br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 57 FORGET<br> 0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE<br> 0.00 0.00 us 0.00 us 0.00 us 19 RELEASEDIR<br> 0.00 30.60 us 27.30 us 39.71 us 5 GETXATTR<br> 0.06 17.42 us 12.10 us 61.92 us 870 ENTRYLK<br> 0.07 40.77 us 36.20 us 58.29 us 435 FTRUNCATE<br> 0.08 45.15 us 38.82 us 68.36 us 435 FALLOCATE<br> 0.13 13.22 us 9.49 us 79.73 us 2436 FLUSH<br> 0.21 117.47 us 98.17 us 169.38 us 435 CREATE<br> 1.47 176.64 us 26.93 us 19510.96 us 2000 OPEN<br> 1.78 61.78 us 25.32 us 17795.12 us 6929 LOOKUP<br> 4.98 19.05 us 10.17 us 7635.73 us 63019 FINODELK<br> 11.53 6369.75 us 392.09 us 16146.95 us 436 FSYNC<br> 17.82 68.11 us 44.52 us 11326.21 us 63019 FXATTROP<br> 61.85 47.11 us 30.86 us 3458.30 us 316194 WRITE<br><br> Duration: 3481 seconds<br> Data Read: 0 bytes<br>Data Written: 46685782016 bytes<br><br>Interval 0 Stats:<br> Block Size: 16384b+<br> No. of Reads: 0<br>No. of Writes: 2849474<br> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br> --------- ----------- ----------- ----------- ------------ ----<br> 0.00 0.00 us 0.00 us 0.00 us 57 FORGET<br> 0.00 0.00 us 0.00 us 0.00 us 4133 RELEASE<br> 0.00 0.00 us 0.00 us 0.00 us 19 RELEASEDIR<br> 0.00 30.60 us 27.30 us 39.71 us 5 GETXATTR<br> 0.06 17.42 us 12.10 us 61.92 us 870 ENTRYLK<br> 0.07 40.77 us 36.20 us 58.29 us 435 FTRUNCATE<br> 0.08 45.15 us 38.82 us 68.36 us 435 FALLOCATE<br> 0.13 13.22 us 9.49 us 79.73 us 2436 FLUSH<br> 0.21 117.47 us 98.17 us 169.38 us 435 CREATE<br> 1.47 176.64 us 26.93 us 19510.96 us 2000 OPEN<br> 1.78 61.78 us 25.32 us 17795.12 us 6929 LOOKUP<br> 4.98 19.05 us 10.17 us 7635.73 us 63019 FINODELK<br> 11.53 6369.75 us 392.09 us 16146.95 us 436 FSYNC<br> 17.82 68.11 us 44.52 us 11326.21 us 63019 FXATTROP<br> 61.85 47.12 us 30.86 us 3458.30 us 316194 WRITE<br><br> Duration: 3481 seconds<br> Data Read: 0 bytes<br>Data Written: 46685782016 bytes</font><br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 6, 2020 at 12:49 AM Mohit Agrawal <<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Along with previous tunning suggested by <span style="color:rgb(0,0,0);white-space:pre-wrap">Strahil please configure "</span><span style="color:rgb(139,0,0);white-space:pre-wrap">ssl.cipher-list"</span></div><div><span style="color:rgb(139,0,0);white-space:pre-wrap">to AES128 for specific volume to improve the performance. As you mentioned you</span></div><div><span style="color:rgb(139,0,0);white-space:pre-wrap">have configured SSL on a volume and </span><span style="color:rgb(139,0,0);white-space:pre-wrap">the </span><span style="color:rgb(139,0,0);white-space:pre-wrap">performance is drop in case of SSL. To improve the same please configure the AES128 cipher list, I hope you should get sufficient performance improvement.</span></div><div><span style="color:rgb(139,0,0);white-space:pre-wrap"><br></span></div><div><span style="color:rgb(139,0,0);white-space:pre-wrap">Please share the result of a performance improvement after configuring the cipher</span></div><div><span style="color:rgb(139,0,0);white-space:pre-wrap">AES128 if it is possible for you.</span></div><div><span style="color:rgb(139,0,0);white-space:pre-wrap"><br></span></div><div><span style="color:rgb(139,0,0);white-space:pre-wrap">Thanks,</span></div><div><span style="color:rgb(139,0,0);white-space:pre-wrap">Mohit Agrawal</span></div><div><span style="color:rgb(139,0,0);white-space:pre-wrap"><br></span></div></div>
________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>
</blockquote></div>