<div><div dir="auto">My guess is there is a corruption in vol list or peer list which has lead glusterd to get into a infinite loop of traversing a peer/volume list and CPU to hog up. Again this is a guess and I've not got a chance to take a detail look at the logs and the strace output.</div><div dir="auto"><br></div><div dir="auto">I believe if you get to reboot the node again the problem will disappear.</div><br><div class="gmail_quote"><div>On Tue, 22 Aug 2017 at 20:07, Serkan Çoban <<a href="mailto:cobanserkan@gmail.com">cobanserkan@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">As an addition perf top shows %80 <a href="http://libc-2.12.so" rel="noreferrer" target="_blank">libc-2.12.so</a> __strcmp_sse42 during<br>
glusterd %100 cpu usage<br>
Hope this helps...<br>
<br>
On Tue, Aug 22, 2017 at 2:41 PM, Serkan Çoban <<a href="mailto:cobanserkan@gmail.com" target="_blank">cobanserkan@gmail.com</a>> wrote:<br>
> Hi there,<br>
><br>
> I have a strange problem.<br>
> Gluster version in 3.10.5, I am testing new servers. Gluster<br>
> configuration is 16+4 EC, I have three volumes, each have 1600 bricks.<br>
> I can successfully create the cluster and volumes without any<br>
> problems. I write data to cluster from 100 clients for 12 hours again<br>
> no problem. But when I try to reboot a node, glusterd process hangs on<br>
> %100 CPU usage and seems to do nothing, no brick processes come<br>
> online. You can find strace of glusterd process for 1 minutes here:<br>
><br>
> <a href="https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0" rel="noreferrer" target="_blank">https://www.dropbox.com/s/c7bxfnbqxze1yus/gluster_strace.out?dl=0</a><br>
><br>
> Here is the glusterd logs:<br>
> <a href="https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0" rel="noreferrer" target="_blank">https://www.dropbox.com/s/hkstb3mdeil9a5u/glusterd.log?dl=0</a><br>
><br>
><br>
> By the way, reboot of one server completes without problem if I reboot<br>
> the servers before creating any volumes.<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div><div dir="ltr">-- <br></div><div class="gmail_signature" data-smartmail="gmail_signature">- Atin (atinm)</div>