<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 14, 2017 at 3:35 PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
This seems to have cleared itself. For future reference though, what kinds of things should I look at do diagnose an issue like this?<br></blockquote><div><br></div><div><br></div><div>Turning on gluster volume profile [1] and sampling the output of profile info at periodic intervals would help. In addition you could also strace the glusterfsd process and/or use `perf record` to determine what the process is doing.</div><div><br></div><div>HTH,</div><div>Vijay</div><div><br></div><div>[1] <a href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/">https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/</a></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks<div class="gmail-m_-5719172739587916076HOEnZb"><div class="gmail-m_-5719172739587916076h5"><br>
<br>
<br>
On 04/14/2017 01:16 PM, Pat Haley wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Hi,<br>
<br>
Today we suddenly experienced a performance drop in gluster: e.g. doing an "ls" of a directory with about 20 files takes about 5 minutes. This is way beyond (and seem separate from) some previous concerns we had.<br>
<br>
Our gluster filesystem is two bricks hosted on a single server. Logging onto that server and doing "top" shows a load average of ~30. In general, no process is showing significant CPU usage except an occasional flash a 3300% from glusterfsd. The rest of our system is not doing any exceptional data demands on the file system (i.e. we aren't suddenly running more jobs than we were yesterday).<br>
<br>
Any thoughts on how we can proceed with debugging this will be greatly appreciated.<br>
<br>
Some additional information:<br>
<br>
glusterfs 3.7.11 built on Apr 27 2016 14:09:22<br>
CentOS release 6.8 (Final)<br>
<br>
<br>
[root@mseas-data2 ~]# gluster volume status data-volume<br>
Status of volume: data-volume<br>
Gluster process TCP Port RDMA Port Online Pid<br>
------------------------------<wbr>------------------------------<wbr>------------------ <br>
Brick mseas-data2:/mnt/brick1 49154 0 Y 5021<br>
Brick mseas-data2:/mnt/brick2 49155 0 Y 5026<br>
<br>
Task Status of Volume data-volume<br>
------------------------------<wbr>------------------------------<wbr>------------------ <br>
Task : Rebalance<br>
ID : 892d9e3a-b38c-4971-b96a-8e4a49<wbr>6685ba<br>
Status : completed<br>
<br>
<br>
[root@mseas-data2 ~]# gluster volume info data-volume<br>
<br>
Volume Name: data-volume<br>
Type: Distribute<br>
Volume ID: c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18<br>
Status: Started<br>
Number of Bricks: 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: mseas-data2:/mnt/brick1<br>
Brick2: mseas-data2:/mnt/brick2<br>
Options Reconfigured:<br>
diagnostics.brick-sys-log-leve<wbr>l: WARNING<br>
performance.readdir-ahead: on<br>
nfs.disable: on<br>
nfs.export-volumes: off<br>
<br>
<br>
</blockquote>
<br>
-- <br>
<br>
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-<br>
Pat Haley Email: <a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a><br>
Center for Ocean Engineering Phone: <a href="tel:%28617%29%20253-6824" value="+16172536824" target="_blank">(617) 253-6824</a><br>
Dept. of Mechanical Engineering Fax: <a href="tel:%28617%29%20253-8125" value="+16172538125" target="_blank">(617) 253-8125</a><br>
MIT, Room 5-213 <a href="http://web.mit.edu/phaley/www/" rel="noreferrer" target="_blank">http://web.mit.edu/phaley/www/</a><br>
77 Massachusetts Avenue<br>
Cambridge, MA 02139-4301<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</div></div></blockquote></div><br></div></div>