<div dir="ltr">I ran the smallfile utility like below for create and read and LOOKUP number looks too high<div><br></div><div># smallfile_cli.py --top /data/shared/tmp/ --threads 8 --files 2000 --file-size 15 --record-size 15 --fsync N --operation create<br></div><div># smallfile_cli.py --top /data/shared/tmp/ --threads 8 --files 2000 --file-size 15 --record-size 15 --fsync N --operation read<br></div><div><br></div><div>total threads = 8<br>total files = 16000<br>total IOPS = 125<br>total data = 0.229 GiB<br>100.00% of requested files processed, warning threshold is 70.00<br>elapsed time = 127.516<br>files/sec = 125.931253<br>IOPS = 125.931253<br>MiB/sec = 1.844696<br></div><div><br></div><div># gluster volume profile atlassian info cumulative | egrep 'Brick|LOOKUP'<br>Brick: node1:/data/foo/gluster<br> 37.43 1424.67 us 13.78 us 2244438.29 us 2024515 LOOKUP<br>Brick: note2:/data/foo/gluster<br> 53.01 413.87 us 14.03 us 262160.52 us 3122542 LOOKUP<br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 26, 2020 at 2:59 PM <<a href="mailto:vadud3@gmail.com">vadud3@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I made the following changes for small file performance as suggested by <a href="http://blog.gluster.org/gluster-tiering-and-small-file-performance/" target="_blank">http://blog.gluster.org/gluster-tiering-and-small-file-performance/</a><div><br></div><div>I am still seeing du -sh /data/shared taking 39 minutes.</div><div><br></div><div>Any other tuning I can do. Most of my files are 15K. Here is sample of small files with size and number of occurrences</div><div><br></div><div>FileSize. # of occurrence</div><div>==== ============</div><div><pre style="box-sizing:border-box;overflow:auto;font-size:14px;padding:1px 0px;margin-top:0px;margin-bottom:0px;line-height:inherit;color:rgb(0,0,0);word-break:break-all;border:0px;border-radius:0px;white-space:pre-wrap;vertical-align:baseline">1.1K 1122
1.1M 1040
1.2K 1281
1.2M 1357
1.3K 1149
1.3M 1098
1.4K 1119
1.5K 1189
1.6K 1036
1.7K 1169
11K 2157
12K 2398
13K 2402
14K 2406
<b>15K 2426</b>
16K 2386
17K 1986
18K 2037
19K 1829
2.0K 1027
2.1K 1048
2.4K 1013
20K 1585
21K 1713
22K 1590
23K 1371
24K 1428
25K 1444
26K 1391
27K 1217
28K 1485
29K 1282
30K 1303
31K 1275
32K 1296
33K 1058
36K 1023
37K 1107
39K 1092
41K 1034
42K 1187
46K 1030</pre><pre style="box-sizing:border-box;overflow:auto;font-size:14px;padding:1px 0px;margin-top:0px;margin-bottom:0px;line-height:inherit;color:rgb(0,0,0);word-break:break-all;border:0px;border-radius:0px;white-space:pre-wrap;vertical-align:baseline"><br></pre></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 25, 2020 at 5:30 PM <<a href="mailto:vadud3@gmail.com" target="_blank">vadud3@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="box-sizing:border-box;font-family:Menlo,Monaco,Consolas,"Courier New",Courier,monospace;font-size:14px;white-space:pre-wrap"><pre>time du -sh /data/shared
431G /data/shared
real 45m49.992s
user 0m20.043s
sys 2m32.456s</pre><pre><br></pre><pre>gluster fs is extremely slow</pre><pre>Any suggestions on what settings to change to improve it?</pre></div><div><br></div>-- <br><div dir="ltr">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div>