<div dir="ltr"><div><div><div>Hi Vijay<br><br></div>Did you manage to look into the gluster profile logs ?<br><br></div>Thanks<br><br></div>Kashif <br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 12, 2017 at 11:40 AM, mohammad kashif <span dir="ltr"><<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Vijay<br><br></div>I have enabled client profiling and used this script <a href="https://github.com/bengland2/gluster-profile-analysis/blob/master/gvp-client.sh" target="_blank">https://github.com/bengland2/<wbr>gluster-profile-analysis/blob/<wbr>master/gvp-client.sh</a> to extract data. I am attaching output files. I don't have any reference data to compare with my output. Hopefully you can make some sense out of it. <br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jun 10, 2017 at 10:47 AM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Would it be possible for you to turn on client profiling and then run du? Instructions for turning on client profiling can be found at [1]. Providing the client profile information can help us figure out where the latency could be stemming from.<div><br></div><div>Regards,</div><div>Vijay</div><div><br></div><div>[1] <a href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Performance%20Testing/#client-side-profiling" target="_blank">https://gluster.readthedoc<wbr>s.io/en/latest/Administrator%<wbr>20Guide/Performance%20Testing/<wbr>#client-side-profiling</a></div></div><div class="m_9097318069126555380HOEnZb"><div class="m_9097318069126555380h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 9, 2017 at 7:22 PM, mohammad kashif <span dir="ltr"><<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Hi Vijay<br><br>Thanks for your quick response. I am using gluster 3.8.11 on Centos 7 servers<br>glusterfs-3.8.11-1.el7.x86_64<br><br></div><div>clients are centos 6 but I tested with a centos 7 client as well and results didn't change<br></div><div><br> gluster volume info
Volume Name: atlasglust
<br>Type: Distribute
<br>Volume ID: fbf0ebb8-deab-4388-9d8a-f72261<wbr>8a624b
<br>Status: Started
<br>Snapshot Count: 0
<br>Number of Bricks: 5
<br>Transport-type: tcp
<br>Bricks:
<br>Brick1: pplxgluster01.x.y.z:/glusterat<wbr>las/brick001/gv0
<br>Brick2: pplxgluster02..x.y.z:/glustera<wbr>tlas/brick002/gv0
<br>Brick3: pplxgluster03.x.y.z:/glusterat<wbr>las/brick003/gv0
<br>Brick4: pplxgluster04.x.y.z:/glusterat<wbr>las/brick004/gv0
<br>Brick5: pplxgluster05.x.y.z:/glusterat<wbr>las/brick005/gv0
<br>Options Reconfigured:
<br>nfs.disable: on
<br>performance.readdir-ahead: on
<br>transport.address-family: inet
<br>auth.allow: x.y.z<br><br></div>I am not using directory quota.<br><br></div><div>Please let me know if you require some more info<br><br></div>Thanks<span class="m_9097318069126555380m_8751817328772499855HOEnZb"><font color="#888888"><br><br></font></span></div><span class="m_9097318069126555380m_8751817328772499855HOEnZb"><font color="#888888">Kashif<br><div><div><br><div><br></div></div></div></font></span></div><div class="m_9097318069126555380m_8751817328772499855HOEnZb"><div class="m_9097318069126555380m_8751817328772499855h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 9, 2017 at 2:34 PM, Vijay Bellur <span dir="ltr"><<a href="mailto:vbellur@redhat.com" target="_blank">vbellur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Can you please provide more details about your volume configuration and the version of gluster that you are using?<div><br></div><div>Regards,</div><div>Vijay</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_9097318069126555380m_8751817328772499855m_8917335311441808956h5">On Fri, Jun 9, 2017 at 5:35 PM, mohammad kashif <span dir="ltr"><<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_9097318069126555380m_8751817328772499855m_8917335311441808956h5"><div dir="ltr"><div><div><div><div>Hi<br><br></div>I have just moved our 400 TB HPC storage from lustre to gluster. It is part of a research institute and users have very small files to big files ( few KB to 20GB) . Our setup consists of 5 servers, each with 96TB RAID 6 disks. All servers are connected through 10G ethernet but not all clients. Gluster volumes are distributed without any replication. There are approximately 80 million files in file system.<br></div><div>I am mounting using glusterfs on clients.<br></div><div><br></div>I have copied everything from lustre to gluster but old file system exist so I can compare.<br><br></div>The problem, I am facing is extremely slow du on even a small directory. Also the time taken is substantially different each time. <br></div>I tried du from same client on a particular directory twice and got these results. <br><div><div><br> time du -sh /data/aa/bb/cc
<br>3.7G /data/aa/bb/cc
<br>real 7m29.243s
<br>user 0m1.448s
<br>sys 0m7.067s
<br><br>time du -sh /data/aa/bb/cc
<br>3.7G /data/aa/bb/cc
<br>real 16m43.735s
<br>user 0m1.097s
<br>sys 0m5.802s<br><br></div><div>16m and 7m is too long for a 3.7 G directory. I must mention that the directory contains huge number of files (208736)<br><br></div><div>but running du on same directory on old data gives this result<br><br>time du -sh /olddata/aa/bb/cc
<br>4.0G /olddata/aa/bb/cc<br>real 3m1.255s
<br>user 0m0.755s
<br>sys 0m38.099s<br><br></div><div>much better if I run same command again<br><br> time du -sh /olddata/aa/bb/cc
<br>4.0G /olddata/aa/bb/cc
<br>real 0m8.309s
<br>user 0m0.313s
<br>sys 0m7.755s
</div><div><br></div><div><div>Is there anything I can do to improve this performance? I would also like hear from some one who is running same kind of setup.<br><br></div><div>Thanks<span class="m_9097318069126555380m_8751817328772499855m_8917335311441808956m_828990225035376178HOEnZb"><font color="#888888"><br><br></font></span></div><span class="m_9097318069126555380m_8751817328772499855m_8917335311441808956m_828990225035376178HOEnZb"><font color="#888888"><div>Kashif <br></div><div><br><br></div></font></span></div></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>