<div dir="ltr"><div>Hi,</div><div> </div><div> Kindly file a bugzilla with proper reproducer, I will update the same on bugzilla after reproducing the same.</div><div><br></div><div>Thanks</div><div>Mohit Agrawal</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 2, 2018 at 7:45 AM, Storage, Dev (Nokia - Global) <span dir="ltr">&lt;<a href="mailto:dev.storage@nokia.com" target="_blank">dev.storage@nokia.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">





<div lang="ZH-CN" link="blue" vlink="purple">
<div class="m_858399917842100374WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thanks for your update!<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">So do you confirm it is an issue and  should add the option set in “glusterd_brick_connect” for this issue, right?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And need we open a Bugzilla record for this issue and make a patch for it?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Best Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Mohit Agrawal [mailto:<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>]
<br>
<b>Sent:</b> Saturday, April 28, 2018 8:05 PM<br>
<b>To:</b> Lian, George (NSB - CN/Hangzhou) &lt;<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>&gt;<br>
<b>Cc:</b> Atin Mukherjee; Storage, Dev (Nokia - Global) &lt;<a href="mailto:dev.storage@nokia.com" target="_blank">dev.storage@nokia.com</a>&gt;; Milind Changire &lt;<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>&gt;; Gaurav Yadav &lt;<a href="mailto:gyadav@redhat.com" target="_blank">gyadav@redhat.com</a>&gt;; Raghavendra Gowdappa &lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;; Samikshan Bairagya &lt;<a href="mailto:sbairagy@redhat.com" target="_blank">sbairagy@redhat.com</a>&gt;; <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a><br>
<b>Subject:</b> Re: glusterfsd start twice leading to glustershd can not connect with glusterfsd<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"> I think this(socket_connect) is not a right place to change the same.If we don&#39;t want to ignore ENOENT(for glusterd) at the time of connecting with the brick<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">process then we can set the same in glusterd_brick_connect in option dict, that will be applicable for glusterd not for every client.<u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Regards<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Mohit Agrawal<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Sat, Apr 28, 2018 at 2:23 PM, Lian, George (NSB - CN/Hangzhou) &lt;<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>&gt; wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Sorry to disturb, mis-spell
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">Atin Mukherjee</span>”<span lang="EN-US"> in last mail, resent!</span></span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Storage,
 Dev (Nokia - Global) <br>
<b>Sent:</b> Saturday, April 28, 2018 4:23 PM<br>
<b>To:</b> Milind Changire &lt;<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>&gt;; Gaurav Yadav &lt;<a href="mailto:gyadav@redhat.com" target="_blank">gyadav@redhat.com</a>&gt;; Mohit Agrawal &lt;<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>&gt;;
 Raghavendra Gowdappa &lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;; Samikshan Bairagya &lt;<a href="mailto:sbairagy@redhat.com" target="_blank">sbairagy@redhat.com</a>&gt;; Atin Mukherjee
<br>
<b>Cc:</b> <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a><br>
<b>Subject:</b> RE: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi, Atin and experts</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">&gt;&gt;&gt;</span><span lang="EN-US"> That would be a hack. I think we need to first figure out on a successful brick connect
 why do we receive a connect, followed by a disconnect event and then another connect event. That&#39;s where the problem exists. I&#39;m waiting to hear for inputs from our RPC experts on this.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">I can answer the above questions now and suppose the root cause of this issue is clear now.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">1)</span><span lang="EN-US" style="font-size:7.0pt;font-family:&quot;Times New Roman&quot;,serif">     
</span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Why receive a connect, followed by a disconnect?</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">In current implement, the connect,</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
 function glusterd_rpc_create called in glusterd_brick_connect not specify option &quot;transport.socket.</span><span lang="EN-US">
</span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">ignore-enoent &quot;.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">So in connect function is
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">socket_connect</span>”<span lang="EN-US"> in socket.c
</span></span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">will use connect instead of connect_loop, in most case it will return ret with -1 and errno=ENOENT, and it will treat as SUCCESS!(attach
 the code below)</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And epoll mechanism will introduce a EPOLLHUP event with a</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
 not really connected socket! So it always trigger a EPOLLHUP handler which will disconnect the socket, and set the start_triggerred flag to faluse for the brick.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">This disconnect issue is 100% reproduced, You could check the glusterd.log to confirm.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">               </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> if (ign_enoent) {</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                       </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> ret = connect_loop (priv-&gt;sock,</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                              <wbr>             </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> SA (&amp;this-&gt;peerinfo.sockaddr),</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                              <wbr>             </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> this-&gt;peerinfo.sockaddr_len);</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                              <wbr>             </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">} else {</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                       </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> ret = connect (priv-&gt;sock,</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                              <wbr>        </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> SA (&amp;this-&gt;peerinfo.sockaddr),</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                              <wbr>        </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> this-&gt;peerinfo.sockaddr_len);</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">               </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> }</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span style="font-size:10.5pt;font-family:DengXian">……</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">if (ret == -1 &amp;&amp; ((errno != EINPROGRESS) &amp;&amp; (errno != ENOENT))) {</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">    </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
</span></i><i><span style="font-size:10.5pt;font-family:DengXian">……………<span lang="EN-US">.</span></span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">               </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> }</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">               </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> else {</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                       </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> /* reset connect_failed so that
 any previous attempts</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                       </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> state is not carried forward */</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                       </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> priv-&gt;connect_failed = 0;</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">                       </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> ret = 0;</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">               </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> }</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">2)</span><span lang="EN-US" style="font-size:7.0pt;font-family:&quot;Times New Roman&quot;,serif">     
</span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Why glusterfsd start twice ?</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">On cluster SN, the brick started with the event ACC event which triggered by friend storage node, it will finally called
 in function glusterd_friend_sm as the following code </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------<wbr>------------------------------<wbr>--------------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">               </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> synclock_unlock (&amp;priv-&gt;big_lock);</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">               </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> glusterd_launch_synctask (glusterd_spawn_daemons,
 NULL);</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">               </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> synclock_lock (&amp;priv-&gt;big_lock);</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="m_858399917842100374m-7482893876161546093msolistparagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt">               </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> glusterd_do_quorum_action ();</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------<wbr>------------------------------<wbr>---------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">From the above code, the brick start maybe trigger twice, one is glusterd_spawn_saemons, another is glusterd_do_quorum_action.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And it is protected by the flag
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">brickinfo-&gt;start_triggered</span>”<span lang="EN-US">, which used to prevent one brick started twice.</span></span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">BUT, when the disconnect issue is always be there as analyzed as the above, it will be occasionally destroy the
 protect mechanism, </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">which is to say, before the second
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">brick start</span>”<span lang="EN-US"> begin, the first
</span>“<span lang="EN-US">brick start</span>”<span lang="EN-US"> on same brick has EPOLLHUP event arrived, and the flag
</span>“<span lang="EN-US">brickinfo-&gt;start_triggered</span>”<span lang="EN-US"> has change to false , then the second
</span>“<span lang="EN-US">brick start</span>”<span lang="EN-US"> will treat it as a fresh brick process, which we has find in the log of
</span>“<span lang="EN-US">glusterd.log</span>”<span lang="EN-US">, abstract one of example as the below:</span></span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.5pt">      </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> Line 632: [2018-04-15 18:06:00.277399] I [glusterd-utils.c:5928:<wbr>glusterd_brick_start]
 0-management: starting a fresh brick process for brick /mnt/bricks/log/brick</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.5pt">      </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> Line 661: [2018-04-15 18:06:00.764239] I [glusterd-utils.c:5928:<wbr>glusterd_brick_start]
 0-management: starting a fresh brick process for brick /mnt/bricks/log/brick</span></i><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And when this occasional issue happen, the glustershd service will abnormal finally.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">So I suppose it is a defect for current implement, and a suggest change solution is one code change in function
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">socket_connect</span>”<span lang="EN-US"> in socket.c as the below:</span></span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">/******</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">              </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> ign_enoent = dict_get_str_boolean (this-&gt;options,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">                  </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> &quot;transport.socket.ignore-<wbr>enoent&quot;, _gf_false);</span><span lang="EN-US" style="font-size:10.5pt">  </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Change to ****/</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">              </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> ign_enoent = dict_get_str_boolean (this-&gt;options,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt">                  </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> &quot;transport.socket.ignore-<wbr>enoent&quot;, _gf_true);</span><span lang="EN-US" style="font-size:10.5pt">  </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">After test, the disconnect issue will not happen anymore!</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">@all expert from gluster,
</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">could you please review my investigation as the above, and share your comments?</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Also, maybe my change will lead to other issue, so could you please give the right solutions?</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thanks &amp; Best Regards</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Atin
 Mukherjee [<a href="mailto:amukherj@redhat.com" target="_blank">mailto:amukherj@redhat.com</a>]
<br>
<b>Sent:</b> Thursday, December 14, 2017 12:37 PM<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) &lt;<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>&gt;<br>
<b>Cc:</b> Milind Changire &lt;<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>&gt;; Gaurav Yadav &lt;<a href="mailto:gyadav@redhat.com" target="_blank">gyadav@redhat.com</a>&gt;; Mohit Agrawal &lt;<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>&gt;;
 Raghavendra Gowdappa &lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;; Samikshan Bairagya &lt;<a href="mailto:sbairagy@redhat.com" target="_blank">sbairagy@redhat.com</a>&gt;<br>
<b>Subject:</b> Re: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">That would be a hack. I think we need to first figure out on a successful brick connect why do we receive a connect, followed by a disconnect event and then another
 connect event. That&#39;s where the problem exists. I&#39;m waiting to hear for inputs from our RPC experts on this.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Thu, Dec 14, 2017 at 8:20 AM, Zhou, Cynthia (NSB - CN/Hangzhou) &lt;<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>&gt;
 wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Hi
</span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">Mukherjee,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">I will try to add some sleep in glusterd_volume_start_<wbr>glusterfs, before it execute glusterd_brick_connect,
 to see if this could help to avoid the “unexpected disconnect event”, I think if this works this race condition could be avoided.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">(周琳)</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">MBB SM HETRAN SW3 MATRIX 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Storage        
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Zhou,
 Cynthia (NSB - CN/Hangzhou) <br>
<b>Sent:</b> Wednesday, December 13, 2017 4:18 PM<br>
<b>To:</b> &#39;Atin Mukherjee&#39; &lt;<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>&gt;; Milind Changire &lt;<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>&gt;<br>
<b>Cc:</b> Gaurav Yadav &lt;<a href="mailto:gyadav@redhat.com" target="_blank">gyadav@redhat.com</a>&gt;; Mohit Agrawal &lt;<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>&gt;; Raghavendra Gowdappa &lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;;
 Samikshan Bairagya &lt;<a href="mailto:sbairagy@redhat.com" target="_blank">sbairagy@redhat.com</a>&gt;<br>
<b>Subject:</b> RE: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">@</span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Mukherjee
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">When I check the code , I find glusterd_volume_start_<wbr>glusterfs will call glusterd_brick_connect, is
 it possible that the glusterd process try to connect to glusterfsd process too early so that it will receive disconnect event?</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">(周琳)</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">MBB SM HETRAN SW3 MATRIX 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Storage        
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Atin
 Mukherjee [mailto:<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>]
<br>
<b>Sent:</b> Tuesday, December 12, 2017 9:00 PM<br>
<b>To:</b> Milind Changire &lt;<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>&gt;<br>
<b>Cc:</b> Gaurav Yadav &lt;<a href="mailto:gyadav@redhat.com" target="_blank">gyadav@redhat.com</a>&gt;; Mohit Agrawal &lt;<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>&gt;; Raghavendra Gowdappa &lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;;
 Samikshan Bairagya &lt;<a href="mailto:sbairagy@redhat.com" target="_blank">sbairagy@redhat.com</a>&gt;; Zhou, Cynthia (NSB - CN/Hangzhou) &lt;<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>&gt;<br>
<b>Subject:</b> Re: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, 12 Dec 2017 at 17:43, Milind Changire &lt;<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>&gt; wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span lang="EN-US">@Cynthia:<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">What is the command(s) used to restart gluster processes ?<u></u><u></u></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">pkill gluster; glusterd is what I did to simulate this.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, Dec 12, 2017 at 3:25 PM, Atin Mukherjee &lt;<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>&gt; wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">Looks like we have a race here. I can see a brick disconnect (RPC_CLNT_DISCONNECT) event when the brick process was coming up.<u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.488767] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.489020] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/export/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.489450] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.489717] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/log/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.490114] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.490398] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/mstate/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.490785] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">What I can see is every time when I restart all the gluster process (simulating a reboot scenario) I could see respective brick disconnect events being received at glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
#0  __glusterd_brick_rpc_notify (rpc=rpc@entry=0x7fffe8008810, mydata=mydata@entry=<wbr>0x7fffe800e0d0,
<br>
    event=event@entry=RPC_CLNT_<wbr>DISCONNECT, data=data@entry=0x0) at glusterd-handler.c:6007<br>
#1  0x00007ffff2936169 in glusterd_big_locked_notify (rpc=0x7fffe8008810, mydata=0x7fffe800e0d0, event=RPC_CLNT_DISCONNECT,
<br>
    data=0x0, notify_fn=0x7ffff2932440 &lt;__glusterd_brick_rpc_notify&gt;) at glusterd-handler.c:68<br>
#2  0x00007ffff78c0343 in rpc_clnt_handle_disconnect (conn=0x7fffe8008840, clnt=0x7fffe8008810) at rpc-clnt.c:876<br>
#3  rpc_clnt_notify (trans=0x7fffe8008ae0, mydata=0x7fffe8008840, event=RPC_TRANSPORT_<wbr>DISCONNECT, data=&lt;optimized out&gt;)<br>
    at rpc-clnt.c:939<br>
#4  0x00007ffff78bca73 in rpc_transport_notify (this=this@entry=<wbr>0x7fffe8008ae0, event=event@entry=RPC_<wbr>TRANSPORT_DISCONNECT,
<br>
    data=data@entry=<wbr>0x7fffe8008ae0) at rpc-transport.c:545<br>
#5  0x00007fffefc252bf in socket_event_poll_err (idx=&lt;optimized out&gt;, gen=&lt;optimized out&gt;, this=0x7fffe8008ae0) at socket.c:1210<br>
#6  socket_event_handler (fd=9, idx=&lt;optimized out&gt;, gen=&lt;optimized out&gt;, data=0x7fffe8008ae0, poll_in=0, poll_out=&lt;optimized out&gt;,
<br>
    poll_err=16) at socket.c:2488<br>
#7  0x00007ffff7b5097c in event_dispatch_epoll_handler (event=0x7fffee9dce84, event_pool=0x64f040) at event-epoll.c:583<br>
#8  event_dispatch_epoll_worker (data=0x70b820) at event-epoll.c:659<br>
#9  0x00007ffff693f36d in start_thread () from /lib64/libpthread.so.0<br>
#10 0x00007ffff61ecbbf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">The question what I have for RPC team is - why do we see a disconnect event in this case?  Any pointers, Mohit/Raghu/Milind?<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">The side effect of this disconnect event is what caused this issue. In __glusterd_brick_rpc_notify we set the brickinfo-&gt;start_triggered to false. So when two
 glusterd_brick_start threads were racing with each other, in between we got a disconnect because of which this flag got reset and we ended up trying to spawn the same brick twice.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, Dec 12, 2017 at 11:05 AM, Zhou, Cynthia (NSB - CN/Hangzhou) &lt;<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>&gt;
 wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Ok. When this happened.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">gluster volume info mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Volume Name: mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Type: Replicate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Volume ID: 5fc29728-b5de-4b5f-8f3c-<wbr>9654acda2577</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Status: Started</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Snapshot Count: 0</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Number of Bricks: 1 x 2 = 2</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Transport-type: tcp</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Bricks:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Brick1: sn-0.local:/mnt/bricks/mstate/<wbr>brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Brick2: sn-1.local:/mnt/bricks/mstate/<wbr>brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Options Reconfigured:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">nfs.disable: on</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">transport.address-family: inet</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">cluster.server-quorum-type: none</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">cluster.consistent-metadata: on</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">server.allow-insecure: on</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">network.ping-timeout: 42</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">performance.client-io-threads: off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">cluster.server-quorum-ratio: 51%</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[root@sn-0:/root]</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">                           
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Also the ps result:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">ps aux | grep gluster</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root       380  0.0  0.6 377892 13760 ?        Ss   20:07   0:00 python /usr/sbin/glustereventsd --pid-file
 /var/run/glustereventsd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root       409  0.0  0.4  82964  9148 ?        S    20:07   0:00 python /usr/sbin/glustereventsd --pid-file
 /var/run/glustereventsd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root       811  0.5  1.0 528720 21636 ?        Ssl  20:07   0:01 /usr/sbin/glusterd --volfile=/opt/nokia/libexec/<wbr>StorageUtils/etc/glusterd/<wbr>glusterd.vol
 -p /run/glusterd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      1088  0.1  0.6 1036648 13996 ?       Ssl  20:07   0:00 /usr/sbin/glusterfsd -s sn-0.local
 --volfile-id export.sn-0.local.mnt-bricks-<wbr>export-brick -p /var/run/gluster/vols/export/<wbr>sn-0.local-mnt-bricks-export-<wbr>brick.pid -S /var/run/gluster/<wbr>44f098f14fe45b74b61c5a1d98993d<wbr>15.socket --brick-name /mnt/bricks/export/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-export-brick.log
 --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49152 --xlator-option export-server.listen-port=<wbr>49152 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      1153  1.8  0.7 1234036 14664 ?       Ssl  20:07   0:06 /usr/sbin/glusterfsd -s sn-0.local
 --volfile-id log.sn-0.local.mnt-bricks-log-<wbr>brick -p /var/run/gluster/vols/log/sn-<wbr>0.local-mnt-bricks-log-brick.<wbr>pid -S /var/run/gluster/<wbr>0af76d9541921cccad34e5fd3041a4<wbr>4d.socket --brick-name /mnt/bricks/log/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-log-brick.log
 --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49156 --xlator-option log-server.listen-port=49156 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      1155  0.0  0.5 772996 10580 ?        Ssl  20:07   0:00 /usr/sbin/glusterfsd -s sn-0.local
 --volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid -S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket --brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log
 --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49154 --xlator-option mstate-server.listen-port=<wbr>49154 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      1167  0.4  0.7 1235512 15488 ?       Ssl  20:07   0:01 /usr/sbin/glusterfsd -s sn-0.local
 --volfile-id services.sn-0.local.mnt-<wbr>bricks-services-brick -p /var/run/gluster/vols/<wbr>services/sn-0.local-mnt-<wbr>bricks-services-brick.pid -S /var/run/gluster/<wbr>fc16d9fe6fd0695555a5592343da7b<wbr>a3.socket --brick-name /mnt/bricks/services/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-services-brick.log
 --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49155 --xlator-option services-server.listen-port=<wbr>49155 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      1188  0.0  0.5 644916 11896 ?        Ssl  20:07   0:00 /usr/sbin/glusterfs -s sn-0.local --volfile-id
 gluster/glustershd -p /var/run/gluster/glustershd/<wbr>glustershd.pid -l /var/log/glusterfs/glustershd.<wbr>log -S /var/run/gluster/<wbr>c405d28a3a79d88436a826d75cf715<wbr>3a.socket --xlator-option *replicate*.node-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      2324  0.0  0.6 629784 13932 ?        Ssl  20:07   0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local
 --volfile-server=sn-1.local --volfile-id=/services /mnt/services</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      2422  0.0  0.5 431036 10604 ?        Ssl  20:07   0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local
 --volfile-server=sn-1.local --volfile-id=/export /mnt/export</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      2520  0.0  0.5 498712 11024 ?        Ssl  20:07   0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local
 --volfile-server=sn-1.local --volfile-id=/log /mnt/log</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      2618  0.0  0.5 431036 10476 ?        Ssl  20:07   0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local
 --volfile-server=sn-1.local --volfile-id=/mstate /mnt/mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      8136  0.0  0.0   8224   784 pts/0    S+   20:13   0:00 grep gluster</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[root@sn-0:/root]</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">#</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">(周琳)</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">MBB SM HETRAN SW3 MATRIX 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Storage        
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Atin
 Mukherjee [mailto:</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">amukherj@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">]
<br>
<b>Sent:</b> Tuesday, December 12, 2017 1:32 PM<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) &lt;</span><span lang="EN-US"><a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">cynthia.zhou@nokia-sbell.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;<br>
<b>Cc:</b> Gaurav Yadav &lt;</span><span lang="EN-US"><a href="mailto:gyadav@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">gyadav@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;;
 Samikshan Bairagya &lt;</span><span lang="EN-US"><a href="mailto:sbairagy@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">sbairagy@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;<br>
<b>Subject:</b> Re: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">I was looking for output of gluster volume info to understand what all volume tunables were turned on.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, Dec 12, 2017 at 10:56 AM, Zhou, Cynthia (NSB - CN/Hangzhou) &lt;<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>&gt;
 wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Hi,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">From our test case when this issue reproduced:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">gluster volume status mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Status of volume: mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Gluster process                       <wbr>      TCP Port  RDMA Port  Online  Pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">------------------------------<wbr>------------------------------<wbr>------------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Brick sn-0.local:/mnt/bricks/mstate/<wbr>brick   49157     0          Y       1155
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Brick sn-1.local:/mnt/bricks/mstate/<wbr>brick   49154     0          Y       1201
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Self-heal Daemon on localhost               N/A       N/A        Y       1188
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Self-heal Daemon on sn-2.local              N/A       N/A        Y       27109</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Self-heal Daemon on sn-1.local              N/A       N/A        Y       4836
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Task Status of Volume mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">------------------------------<wbr>------------------------------<wbr>------------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">There are no active volume tasks</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[root@sn-0:/root]</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">#</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">(周琳)</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">MBB SM HETRAN SW3 MATRIX 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Storage        
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Atin
 Mukherjee [mailto:</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">amukherj@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">]
<br>
<b>Sent:</b> Tuesday, December 12, 2017 12:02 PM<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) &lt;</span><span lang="EN-US"><a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">cynthia.zhou@nokia-sbell.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;<br>
<b>Cc:</b> Gaurav Yadav &lt;</span><span lang="EN-US"><a href="mailto:gyadav@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">gyadav@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;;
 Samikshan Bairagya &lt;</span><span lang="EN-US"><a href="mailto:sbairagy@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">sbairagy@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;<br>
<b>Subject:</b> Re: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, Dec 12, 2017 at 8:34 AM, Zhou, Cynthia (NSB - CN/Hangzhou) &lt;<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>&gt;
 wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">By the way , when I check the code I find that in</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">glusterd_op_perform_add_bricks and glusterd_start_volume, glusterd_brick_start is not triggered within
 pthread_mutex_lock, do you think we could add pthread_mutex_lock?</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I don&#39;t think that&#39;s needed as these functions are triggered through CLI command. We get into this race when the brick start is triggered asynchronously from
 a restart glusterd code path. Do you have the volume info output when this issue occurred?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">(周琳)</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">MBB SM HETRAN SW3 MATRIX 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Storage        
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Zhou,
 Cynthia (NSB - CN/Hangzhou) <br>
<b>Sent:</b> Tuesday, December 12, 2017 9:23 AM<br>
<b>To:</b> &#39;Gaurav Yadav&#39; &lt;</span><span lang="EN-US"><a href="mailto:gyadav@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">gyadav@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;;
 Atin Mukherjee &lt;</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">amukherj@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;<br>
<b>Cc:</b> Samikshan Bairagya &lt;</span><span lang="EN-US"><a href="mailto:sbairagy@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">sbairagy@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;<br>
<b>Subject:</b> RE: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">HI,
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Thanks for responding!</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">My testing version has include the correction of bug-1506589, I’ve also checked my gluster source code.
 so this is a new issue.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Unfortunately this issue rarely appear, It only appeared once until now. it is found after reboot sn
 node.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Enclosed is the content in /var/lib/glusterd and /var/log/glusterfs</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">The time when this issue happened is , only mstate volume has this issue.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[<span style="color:red">2017-12-09 18:07:02.743046</span>] W [glusterfsd.c:1375:cleanup_<wbr>and_exit] (--&gt;/lib64/libpthread.so.0(+<wbr>0x7454)
 [0x7f5df4db6454] --&gt;/usr/sbin/glusterfsd(<wbr>glusterfs_sigwaiter+0xdc) [0x40a50c] --&gt;/usr/sbin/glusterfsd(<wbr>cleanup_and_exit+0x77) [0x4089ec] ) 0-: received signum (15), shutting down</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[<span style="color:red">2017-12-09 18:07:34.416169</span>] I [MSGID: 100030] [glusterfsd.c:2524:main]
 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.12.3 (args: /usr/sbin/glusterfsd -s sn-0.local --volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid -S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket
 --brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49154 --xlator-option mstate-server.listen-port=<wbr>49154 --xlator-option transport.socket.bind-address=<wbr>sn-0.local)</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif;color:red">[2017-12-09 18:07:34.511598</span><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">]
 I [MSGID: 100030] [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.12.3 (args: /usr/sbin/glusterfsd -s sn-0.local --volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid
 -S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket --brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49157 --xlator-option
 mstate-server.listen-port=<wbr>49157 --xlator-option transport.socket.bind-address=<wbr>sn-0.local)</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">(周琳)</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">MBB SM HETRAN SW3 MATRIX 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Arial&quot;,sans-serif">Storage        
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Gaurav
 Yadav [</span><span lang="EN-US"><a href="mailto:gyadav@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">mailto:gyadav@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">]
<br>
<b>Sent:</b> Monday, December 11, 2017 6:58 PM<br>
<b>To:</b> Atin Mukherjee &lt;</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">amukherj@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;<br>
<b>Cc:</b> Samikshan Bairagya &lt;</span><span lang="EN-US"><a href="mailto:sbairagy@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">sbairagy@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;;
 Zhou, Cynthia (NSB - CN/Hangzhou) &lt;</span><span lang="EN-US"><a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">cynthia.zhou@nokia-sbell.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">&gt;<br>
<b>Subject:</b> Re: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
After seeing logs which you have shared here, it seems it&#39;s a port mismatch issue.
<br>
We had a similar issue <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1506589" target="_blank">
bug-1506589,</a> &quot;Please confirm that you too are seeing same behavior&quot;.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
If not please share setup details along with.. <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">* files from /var/lib/glusterd/ directory,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">* log files from /var/log/glusterfs<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">Gaurav<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Mon, Dec 11, 2017 at 3:16 PM, Atin Mukherjee &lt;<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>&gt; wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Adding couple of my colleagues to the thread since I will be busy through out this week. If this is easily reproducible then could you share the exact steps?<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Mon, 11 Dec 2017 at 14:56, Zhou, Cynthia (NSB - CN/Hangzhou) &lt;<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>&gt;
 wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Hi,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">        Good day!</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif">       
</span><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">I see your change
</span><span lang="EN-US" style="font-size:8.5pt;font-family:&quot;Courier New&quot;">I292f1e58d6971e111725e1baea1fe<wbr>98b890b43e2 i</span><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">n glusterfs code.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">        However from my test glusterfs 3.12.3 (already contain that modification) still has something wrong,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">gluster volume status mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Status of volume: mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Gluster process                       <wbr>      TCP Port  RDMA Port  Online  Pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">------------------------------<wbr>------------------------------<wbr>------------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Brick sn-0.local:/mnt/bricks/mstate/<wbr>brick  
<b><span style="color:red">49157</span></b>     0          Y       1155 </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Brick sn-1.local:/mnt/bricks/mstate/<wbr>brick   49154     0          Y       1201
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Self-heal Daemon on localhost               N/A       N/A        Y       1188
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Self-heal Daemon on sn-2.local              N/A       N/A        Y       27109</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Self-heal Daemon on sn-1.local              N/A       N/A        Y       4836
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Task Status of Volume mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">------------------------------<wbr>------------------------------<wbr>------------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">There are no active volume tasks</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif">       
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">But ps shows</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">ps aux | grep gluster</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root       380  0.0  0.6 377892 13760 ?        Ss   20:07   0:00 python /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root       409  0.0  0.4  82964  9148 ?        S    20:07   0:00 python /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root       811  0.5  1.0 528720 21636 ?        Ssl  20:07   0:01 /usr/sbin/glusterd --volfile=/opt/nokia/libexec/<wbr>StorageUtils/etc/glusterd/<wbr>glusterd.vol -p /run/glusterd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      1088  0.1  0.6 1036648 13996 ?       Ssl  20:07   0:00 /usr/sbin/glusterfsd -s sn-0.local --volfile-id export.sn-0.local.mnt-bricks-<wbr>export-brick -p /var/run/gluster/vols/export/<wbr>sn-0.local-mnt-bricks-export-<wbr>brick.pid
 -S /var/run/gluster/<wbr>44f098f14fe45b74b61c5a1d98993d<wbr>15.socket --brick-name /mnt/bricks/export/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-export-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49152 --xlator-option
 export-server.listen-port=<wbr>49152 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      1153  1.8  0.7 1234036 14664 ?       Ssl  20:07   0:06 /usr/sbin/glusterfsd -s sn-0.local --volfile-id log.sn-0.local.mnt-bricks-log-<wbr>brick -p /var/run/gluster/vols/log/sn-<wbr>0.local-mnt-bricks-log-brick.<wbr>pid
 -S /var/run/gluster/<wbr>0af76d9541921cccad34e5fd3041a4<wbr>4d.socket --brick-name /mnt/bricks/log/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-log-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49156 --xlator-option
 log-server.listen-port=49156 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      1155  0.0  0.5 772996 10580 ?        Ssl  20:07   0:00 /usr/sbin/glusterfsd -s sn-0.local --volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid
 -S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket --brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49154 --xlator-option
 mstate-server.listen-port=<b><span style="color:red">4915<wbr>4</span></b> --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      1167  0.4  0.7 1235512 15488 ?       Ssl  20:07   0:01 /usr/sbin/glusterfsd -s sn-0.local --volfile-id services.sn-0.local.mnt-<wbr>bricks-services-brick -p /var/run/gluster/vols/<wbr>services/sn-0.local-mnt-<wbr>bricks-services-brick.pid
 -S /var/run/gluster/<wbr>fc16d9fe6fd0695555a5592343da7b<wbr>a3.socket --brick-name /mnt/bricks/services/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-services-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49155 --xlator-option
 services-server.listen-port=<wbr>49155 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      1188  0.0  0.5 644916 11896 ?        Ssl  20:07   0:00 /usr/sbin/glusterfs -s sn-0.local --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/<wbr>glustershd.pid -l
 /var/log/glusterfs/glustershd.<wbr>log -S /var/run/gluster/<wbr>c405d28a3a79d88436a826d75cf715<wbr>3a.socket --xlator-option *replicate*.node-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      2324  0.0  0.6 629784 13932 ?        Ssl  20:07   0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local --volfile-server=sn-1.local --volfile-id=/services /mnt/services</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      2422  0.0  0.5 431036 10604 ?        Ssl  20:07   0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local --volfile-server=sn-1.local --volfile-id=/export /mnt/export</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      2520  0.0  0.5 498712 11024 ?        Ssl  20:07   0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local --volfile-server=sn-1.local --volfile-id=/log /mnt/log</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      2618  0.0  0.5 431036 10476 ?        Ssl  20:07   0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local --volfile-server=sn-1.local --volfile-id=/mstate /mnt/mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">root      8136  0.0  0.0   8224   784 pts/0    S+   20:13   0:00 grep gluster</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[root@sn-0:/root]</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<b><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">So glustershd can not connect with mstate brick on sn-0</span></b><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">gluster volume heal mstate info</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Brick sn-0.local:/mnt/bricks/mstate/<wbr>brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<b><span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif;color:red">Status: Transport endpoint is not connected</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Number of entries: -</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Brick sn-1.local:/mnt/bricks/mstate/<wbr>brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Status: Connected</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Number of entries: 0</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">From sn-0 glusterd log I find:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.366851] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/export/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.368484] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.406488] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/log/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.410384] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif;color:red">[2017-12-09 18:07:34.410530] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.413819] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.413926] C [MSGID: 106003] [glusterd-server-quorum.c:352:<wbr>glusterd_do_volume_quorum_<wbr>action] 0-management: Server quorum regained for volume services. Starting local
 bricks.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.414031] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/services/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.417573] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.462163] I [MSGID: 106492] [glusterd-handler.c:2718:__<wbr>glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: e4ea0550-873b-410c-a29e-<wbr>3901c4737cd8</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.470228] I [MSGID: 106502] [glusterd-handler.c:2763:__<wbr>glusterd_handle_friend_update] 0-management: Received my uuid as Friend</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.470432] I [MSGID: 106492] [glusterd-handler.c:2718:__<wbr>glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: feddd74c-c768-4157-b152-<wbr>31d5dd488dcc</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.470493] I [MSGID: 106502] [glusterd-handler.c:2763:__<wbr>glusterd_handle_friend_update] 0-management: Received my uuid as Friend</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.479540] I [MSGID: 106163] [glusterd-handshake.c:1316:__<wbr>glusterd_mgmt_hndsk_versions_<wbr>ack] 0-management: using the op-version 31202</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.488767] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.489020] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/export/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.489450] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.489717] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/log/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.490114] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.490398] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/mstate/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.490785] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.491049] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/<wbr>services/brick has disconnected from
 glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.491514] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__<wbr>glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: feddd74c-c768-4157-b152-<wbr>31d5dd488dcc, host: sn-1.local,
 port: 0</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.501902] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/log/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.503376] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif;color:red">[2017-12-09 18:07:34.503502] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.506655] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.507887] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.508188] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/log/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.509479] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">……</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.023087] I [glusterd-utils.c:5834:<wbr>glusterd_brick_start] 0-management: discovered already-running brick /mnt/bricks/export/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.023106] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/export/brick on port 49152</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.060419] I [glusterd-utils.c:5834:<wbr>glusterd_brick_start] 0-management: discovered already-running brick /mnt/bricks/log/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.060442] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/log/brick on port 49156</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.094987] I [glusterd-utils.c:5834:<wbr>glusterd_brick_start] 0-management: discovered already-running brick /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.095008] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/mstate/brick on port
<span style="color:red">49157</span></span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.129059] I [glusterd-utils.c:5834:<wbr>glusterd_brick_start] 0-management: discovered already-running brick /mnt/bricks/services/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.129108] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/services/brick on port 49155</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.159408] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-snapd: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">……</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.919192] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__<wbr>glusterd_friend_update_cbk] 0-management: Received ACC from uuid: e4ea0550-873b-410c-a29e-<wbr>3901c4737cd8</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.920269] I [MSGID: 106492] [glusterd-handler.c:2718:__<wbr>glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: e4ea0550-873b-410c-a29e-<wbr>3901c4737cd8</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.927413] I [MSGID: 106502] [glusterd-handler.c:2763:__<wbr>glusterd_handle_friend_update] 0-management: Received my uuid as Friend</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.949762] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__<wbr>glusterd_friend_update_cbk] 0-management: Received ACC from uuid: feddd74c-c768-4157-b152-<wbr>31d5dd488dcc</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.949817] I [MSGID: 106492] [glusterd-handler.c:2718:__<wbr>glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: feddd74c-c768-4157-b152-<wbr>31d5dd488dcc</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:36.949882] I [MSGID: 106502] [glusterd-handler.c:2763:__<wbr>glusterd_handle_friend_update] 0-management: Received my uuid as Friend</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:37.768326] W [MSGID: 101095] [xlator.c:213:xlator_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.3/<wbr>xlator/nfs/server.so: cannot open shared object file: No such file
 or directory</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:37.783811] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/services/brick on port 49155</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:37.783901] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/mstate/brick on port
<span style="color:red">49154</span></span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:37.783966] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/log/brick on port 49156</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:37.784032] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/export/brick on port 49152</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:37.810305] W [MSGID: 101095] [xlator.c:162:xlator_volopt_<wbr>dynload] 0-xlator: /usr/lib64/glusterfs/3.12.3/<wbr>xlator/nfs/server.so: cannot open shared object file: No such
 file or directory</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">And the mstate brick process on sn-0 shows:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:02.743046] W [glusterfsd.c:1375:cleanup_<wbr>and_exit] (--&gt;/lib64/libpthread.so.0(+<wbr>0x7454) [0x7f5df4db6454] --&gt;/usr/sbin/glusterfsd(<wbr>glusterfs_sigwaiter+0xdc) [0x40a50c]
 --&gt;/usr/sbin/glusterfsd(<wbr>cleanup_and_exit+0x77) [0x4089ec] ) 0-: received signum (15), shutting down</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.416169] I [MSGID: 100030] [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.12.3 (args: /usr/sbin/glusterfsd -s sn-0.local
 --volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid -S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket --brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log
 --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port
<span style="color:red">49154</span> --xlator-option mstate-server.listen-port=<wbr>49154 --xlator-option transport.socket.bind-address=<wbr>sn-0.local)</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.511598] I [MSGID: 100030] [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.12.3 (args: /usr/sbin/glusterfsd -s sn-0.local
 --volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid -S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket --brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log
 --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port
<span style="color:red">49157</span> --xlator-option mstate-server.listen-port=<wbr>49157 --xlator-option transport.socket.bind-address=<wbr>sn-0.local)</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.542268] E [MSGID: 100018] [glusterfsd.c:2109:glusterfs_<wbr>pidfile_update] 0-glusterfsd: pidfile /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid
 lock failed [Resource temporarily unavailable]</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.549242] W [MSGID: 101002] [options.c:995:xl_opt_<wbr>validate] 0-glusterfs: option &#39;address-family&#39; is deprecated, preferred is &#39;transport.address-family&#39;, continuing
 with correction</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:34.550119] I [MSGID: 101190] [event-epoll.c:613:event_<wbr>dispatch_epoll_worker] 0-epoll: Started thread with index 1</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:37.690037] I [rpcsvc.c:2322:rpcsvc_set_<wbr>outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">[2017-12-09 18:07:37.690125] W [MSGID: 101002] [options.c:995:xl_opt_<wbr>validate] 0-mstate-server: option &#39;listen-port&#39; is deprecated, preferred is &#39;transport.socket.listen-port&#39;<wbr>, continuing
 with correction</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Times New Roman&quot;,serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">……</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">Final graph:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">+-----------------------------<wbr>------------------------------<wbr>-------------------+</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">  1: volume mstate-posix</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">  2:     type storage/posix</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">  3:     option glusterd-uuid f830d809-c935-451f-853a-<wbr>1364f76a4160</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">  4:     option directory /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">  5:     option volume-id 5fc29728-b5de-4b5f-8f3c-<wbr>9654acda2577</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">  6:     option shared-brick-count 1</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">  7: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">  8: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">  9: volume mstate-trash</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">10:     type features/trash</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">11:     option trash-dir .trashcan</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">12:     option brick-path /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">13:     option trash-internal-op off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">14:     subvolumes mstate-posix</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">15: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">16: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">17: volume mstate-changetimerecorder</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">18:     type features/changetimerecorder</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">19:     option db-type sqlite3</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">20:     option hot-brick off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">21:     option db-name brick.db</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">22:     option db-path /mnt/bricks/mstate/brick/.<wbr>glusterfs/</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">23:     option record-exit off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">24:     option ctr_link_consistency off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">25:     option ctr_lookupheal_link_timeout 300</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">26:     option ctr_lookupheal_inode_timeout 300</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">27:     option record-entry on</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">28:     option ctr-enabled off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">29:     option record-counters off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">30:     option ctr-record-metadata-heat off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">31:     option sql-db-cachesize 12500</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">32:     option sql-db-wal-autocheckpoint 25000</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">33:     subvolumes mstate-trash</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">34: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">35: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">36: volume mstate-changelog</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">37:     type features/changelog</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">38:     option changelog-brick /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">39:     option changelog-dir /mnt/bricks/mstate/brick/.<wbr>glusterfs/changelogs</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">40:     option changelog-barrier-timeout 120</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">41:     subvolumes mstate-changetimerecorder</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">42: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">43: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">44: volume mstate-bitrot-stub</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">45:     type features/bitrot-stub</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">46:     option export /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">47:     option bitrot disable</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">48:     subvolumes mstate-changelog</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">49: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">50: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">51: volume mstate-access-control</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">52:     type features/access-control</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">53:     subvolumes mstate-bitrot-stub</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">54: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">55: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">56: volume mstate-locks</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">57:     type features/locks</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">58:     subvolumes mstate-access-control</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">59: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">60: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">61: volume mstate-worm</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">62:     type features/worm</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">63:     option worm off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">64:     option worm-file-level off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">65:     subvolumes mstate-locks</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">66: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">67: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">68: volume mstate-read-only</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">69:     type features/read-only</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">70:     option read-only off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">71:     subvolumes mstate-worm</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">72: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">73: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">74: volume mstate-leases</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">75:     type features/leases</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">76:     option leases off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">77:     subvolumes mstate-read-only</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">78: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">79: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">80: volume mstate-upcall</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">81:     type features/upcall</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">82:     option cache-invalidation off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">83:     subvolumes mstate-leases</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">84: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">85: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">86: volume mstate-io-threads</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">87:     type performance/io-threads</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">88:     subvolumes mstate-upcall</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">89: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">90: 
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">91: volume mstate-selinux</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">92:     type features/selinux</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">93:     option selinux on</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">94:     subvolumes mstate-io-threads</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:&quot;Calibri&quot;,sans-serif">95: end-volume&lt;</span><span class="HOEnZb"><font color="#888888"><span lang="EN-US"><u></u><u></u></span></font></span></p><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
<div>
<p class="MsoNormal"><span lang="EN-US" style="color:#888888">--
<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="color:#888888">- Atin (atinm)<u></u><u></u></span></p>
</div>
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</font></span></div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
</div>
</div>

</blockquote></div><br></div>