<div dir="ltr">- # gluster --version<br>glusterfs 7.5<div><br></div><div>- # gluster volume status atlassian </div>Status of volume: atlassian<br>Gluster process TCP Port RDMA Port Online Pid<br>------------------------------------------------------------------------------<br>Brick node1:/data/atlassian/gluster 49152 0 Y 1791<br>Brick node2:/data/atlassian/gluster 49152 0 Y 1773<br>Self-heal Daemon on localhost N/A N/A Y 1807<br>Self-heal Daemon on node1.example.c<br><a href="http://example.net">example.net</a> N/A N/A Y 1778<br><br>Task Status of Volume atlassian<br>------------------------------------------------------------------------------<br>There are no active volume tasks<div><br></div><div>- # attached pre-du and during-du log from server</div><div><br></div><div><br></div><div>- I do not have a remote client. when I tried to run these</div><div><span style="color:rgba(0,0,0,0.87);font-family:Roboto,-apple-system,system-ui,Helvetica,Arial,sans-serif;font-size:16px">gluster volume profile your-volume start says already started since I am on the server</span><br></div><div># setfattr -n trusted.io-stats-dump -v /tmp/io-stats-pre.txt /mnt runs but no output in /tmp/io-stats-pre.txt<span style="color:rgba(0,0,0,0.87);font-family:Roboto,-apple-system,system-ui,Helvetica,Arial,sans-serif;font-size:16px"><br></span></div><div><br></div><div><br></div><div>- # gluster volume heal atlassian info </div>Brick node1:/data/atlassian/gluster<br>Status: Connected<br>Number of entries: 0<br><br>Brick node2:/data/atlassian/gluster<br>Status: Connected<br>Number of entries: 0<div><br></div><div>Let me know if you need anything else. Appreciate your help</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 27, 2020 at 3:27 AM Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com">ksubrahm@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Please provide the following information to understand the setup and debug this further:</div><div>- Which version of gluster you are using?</div><div>- 'gluster volume status atlassian' to confirm both bricks and shds are up or not</div><div>- Complete output of 'gluster volume profile atlassian info' before running 'du' and during 'du'. Redirect this output to separate files and attach them here</div><div>- Get the client side profile as well by following <a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Performance%20Testing/" target="_blank">https://docs.gluster.org/en/latest/Administrator%20Guide/Performance%20Testing/</a></div><div>- 'gluster volume heal atlassian info' to check whether there are any pending heals and client side heal is contributing to this</div><div><br></div><div>Regards,</div><div>Karthik</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 27, 2020 at 1:06 AM <<a href="mailto:vadud3@gmail.com" target="_blank">vadud3@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I had a parsing error. It is Volume Name: atlassian</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 26, 2020 at 3:12 PM <<a href="mailto:vadud3@gmail.com" target="_blank">vadud3@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"># gluster volume info<br><br>Volume Name: myvol<br>Type: Replicate<br>Volume ID: cbdef65c-79ea-496e-b777-b6a2981b29cf<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 1 x 2 = 2<br>Transport-type: tcp<br>Bricks:<br>Brick1: node1:/data/foo/gluster<br>Brick2: node2:/data/foo/gluster<br>Options Reconfigured:<br>client.event-threads: 4<br>server.event-threads: 4<br>performance.stat-prefetch: on<br>network.inode-lru-limit: 16384<br>performance.md-cache-timeout: 1<br>performance.cache-invalidation: false<br>performance.cache-samba-metadata: false<br>features.cache-invalidation-timeout: 600<br>features.cache-invalidation: on<br>performance.io-thread-count: 16<br>performance.cache-refresh-timeout: 5<br>performance.write-behind-window-size: 5MB<br>performance.cache-size: 1GB<br>transport.address-family: inet<br>storage.fips-mode-rchecksum: on<br>nfs.disable: on<br>performance.client-io-threads: off<br>diagnostics.latency-measurement: on<br>diagnostics.count-fop-hits: on<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, May 26, 2020 at 3:06 PM Sunil Kumar Heggodu Gopala Acharya <<a href="mailto:sheggodu@redhat.com" target="_blank">sheggodu@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>Please share the gluster volume information.</div><div><br></div><div># gluster vol info</div><div><br></div><div><br></div><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr">Regards,<br><p style="font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>Sunil kumar</span> <span>Acharya</span></p></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 27, 2020 at 12:30 AM <<a href="mailto:vadud3@gmail.com" target="_blank">vadud3@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I made the following changes for small file performance as suggested by <a href="http://blog.gluster.org/gluster-tiering-and-small-file-performance/" target="_blank">http://blog.gluster.org/gluster-tiering-and-small-file-performance/</a><div><br></div><div>I am still seeing du -sh /data/shared taking 39 minutes.</div><div><br></div><div>Any other tuning I can do. Most of my files are 15K. Here is sample of small files with size and number of occurrences</div><div><br></div><div>FileSize. # of occurrence</div><div>==== ============</div><div><pre style="box-sizing:border-box;overflow:auto;font-size:14px;padding:1px 0px;margin-top:0px;margin-bottom:0px;line-height:inherit;color:rgb(0,0,0);word-break:break-all;border:0px none;border-radius:0px;white-space:pre-wrap;vertical-align:baseline">1.1K 1122
1.1M 1040
1.2K 1281
1.2M 1357
1.3K 1149
1.3M 1098
1.4K 1119
1.5K 1189
1.6K 1036
1.7K 1169
11K 2157
12K 2398
13K 2402
14K 2406
<b>15K 2426</b>
16K 2386
17K 1986
18K 2037
19K 1829
2.0K 1027
2.1K 1048
2.4K 1013
20K 1585
21K 1713
22K 1590
23K 1371
24K 1428
25K 1444
26K 1391
27K 1217
28K 1485
29K 1282
30K 1303
31K 1275
32K 1296
33K 1058
36K 1023
37K 1107
39K 1092
41K 1034
42K 1187
46K 1030</pre><pre style="box-sizing:border-box;overflow:auto;font-size:14px;padding:1px 0px;margin-top:0px;margin-bottom:0px;line-height:inherit;color:rgb(0,0,0);word-break:break-all;border:0px none;border-radius:0px;white-space:pre-wrap;vertical-align:baseline"><br></pre></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, May 25, 2020 at 5:30 PM <<a href="mailto:vadud3@gmail.com" target="_blank">vadud3@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="box-sizing:border-box;font-family:Menlo,Monaco,Consolas,"Courier New",Courier,monospace;font-size:14px;white-space:pre-wrap"><pre>time du -sh /data/shared
431G /data/shared
real 45m49.992s
user 0m20.043s
sys 2m32.456s</pre><pre><br></pre><pre>gluster fs is extremely slow</pre><pre>Any suggestions on what settings to change to improve it?</pre></div><div><br></div>-- <br><div dir="ltr">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Asif Iqbal<br>PGP Key: 0xE62693C5 KeyServer: <a href="http://pgp.mit.edu" target="_blank">pgp.mit.edu</a><br>A: Because it messes up the order in which people normally read text.<br>Q: Why is top-posting such a bad thing?<br><br></div>