<div dir="ltr">Thank you for the acknowledgement.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 31, 2017 at 8:30 PM, mohammad kashif <span dir="ltr"><<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Hi Atin<br><br></div>Thanks, I was not running any script or gluster command. But now gluster status command started working. CPU usage also came down and looking at the ganglia graph, cpu usage is strongly correlated with network activity. <br><br></div>It may be due to high load that status command failed. I will keep an eye and see whether it happens again.<br><br></div>Cheers<span class="HOEnZb"><font color="#888888"><br><br></font></span></div><span class="HOEnZb"><font color="#888888">Kashif <br></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 31, 2017 at 2:40 AM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br><div class="gmail_quote"><span><div dir="auto">On Wed, 30 Aug 2017 at 20:55, mohammad kashif <<a href="mailto:kashif.alig@gmail.com" target="_blank">kashif.alig@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div><div><div><div>Hi<br><br></div>I am running a 400TB five node purely distributed gluster setup. I am troubleshooting an issue where some times files creation fails. I found that volume status is not working<br><br>gluster volume status
<br>Another transaction is in progress for atlasglust. Please try again after sometime.<br><br></div>When I tried from other node then it seems two nodes have Locking issue<br><br>gluster volume status
<br>Locking failed on pplxgluster01... Please check log file for details.
<br>Locking failed on pplxgluster04... Please check log file for details.</div></div></div></div></blockquote><div dir="auto"><br></div></span><div dir="auto">This suggests that there are concurrent gluster cli operations been performed on the same volume. Are you monitoring the cluster through nagios or you have a script on all the nodes which checks for volume's health in a period of interval? Please note glusterd will process one cli operation on a volume at one time, the rest all transactions on the same volume will be failed.</div><div dir="auto"><br></div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span><div><div><div><div><br><br></div>Also noticed that glusterfsd process is using around 1000% cpu usage. It is a decent server with 16 core and 64GB RAM.<br><br></div>Gluster version is 3.11.2-1 <br><br></div>Can you please suggest that how to troubleshoot further?<br><div><div><div><div><br></div><div>Thanks</div><div><br></div><div>Kashif<br></div><div> <br>
</div></div></div></div></div></span><span>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></span></blockquote></div></div><span class="m_6879112604004757027HOEnZb"><font color="#888888"><div dir="ltr">-- <br></div><div class="m_6879112604004757027m_7578140280252112303gmail_signature" data-smartmail="gmail_signature">- Atin (atinm)</div>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div>