<div><br><div class="gmail_quote"><div dir="auto">On Wed, 30 Aug 2017 at 20:55, mohammad kashif <<a href="mailto:kashif.alig@gmail.com">kashif.alig@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div><div><div><div>Hi<br><br></div>I am running a 400TB five node purely distributed gluster setup. I am troubleshooting an issue where some times files creation fails. I found that volume status is not working<br><br>gluster volume status
<br>Another transaction is in progress for atlasglust. Please try again after sometime.<br><br></div>When I tried from other node then it seems two nodes have Locking issue<br><br>gluster volume status
<br>Locking failed on pplxgluster01... Please check log file for details.
<br>Locking failed on pplxgluster04... Please check log file for details.</div></div></div></div></blockquote><div dir="auto"><br></div><div dir="auto">This suggests that there are concurrent gluster cli operations been performed on the same volume. Are you monitoring the cluster through nagios or you have a script on all the nodes which checks for volume's health in a period of interval? Please note glusterd will process one cli operation on a volume at one time, the rest all transactions on the same volume will be failed.</div><div dir="auto"><br></div><div dir="auto"><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div><div><br><br></div>Also noticed that glusterfsd process is using around 1000% cpu usage. It is a decent server with 16 core and 64GB RAM.<br><br></div>Gluster version is 3.11.2-1 <br><br></div>Can you please suggest that how to troubleshoot further?<br><div><div><div><div><br></div><div>Thanks</div><div><br></div><div>Kashif<br></div><div> <br>
</div></div></div></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div><div dir="ltr">-- <br></div><div class="gmail_signature" data-smartmail="gmail_signature">- Atin (atinm)</div>