<div dir="ltr"><div>Hi,</div><div><br></div> I think this(socket_connect) is not a right place to change the same.If we don't want to ignore ENOENT(for glusterd) at the time of connecting with the brick<div>process then we can set the same in glusterd_brick_connect in option dict, that will be applicable for glusterd not for every client.<div><div><br></div><div>Regards</div><div>Mohit Agrawal</div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Apr 28, 2018 at 2:23 PM, Lian, George (NSB - CN/Hangzhou) <span dir="ltr"><<a href="mailto:george.lian@nokia-sbell.com" target="_blank">george.lian@nokia-sbell.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="ZH-CN" link="blue" vlink="purple">
<div class="m_-7482893876161546093WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Sorry to disturb, mis-spell “Atin Mukherjee” in last mail, resent!<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Storage, Dev (Nokia - Global)
<br>
<b>Sent:</b> Saturday, April 28, 2018 4:23 PM<br>
<b>To:</b> Milind Changire <<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>>; Gaurav Yadav <<a href="mailto:gyadav@redhat.com" target="_blank">gyadav@redhat.com</a>>; Mohit Agrawal <<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>>; Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>; Samikshan Bairagya <<a href="mailto:sbairagy@redhat.com" target="_blank">sbairagy@redhat.com</a>>; Atin Mukherjee
<br>
<b>Cc:</b> <a href="mailto:gluster-devel@gluster.org" target="_blank">gluster-devel@gluster.org</a><br>
<b>Subject:</b> RE: glusterfsd start twice leading to glustershd can not connect with glusterfsd<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi, Atin and experts<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">>>></span><span lang="EN-US"> That would be a hack. I think we need to first figure out on a successful brick connect why do we receive a connect, followed by a disconnect
event and then another connect event. That's where the problem exists. I'm waiting to hear for inputs from our RPC experts on this.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">I can answer the above questions now and suppose the root cause of this issue is clear now.<u></u><u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt">
<u></u><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><span>1)<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Why receive a connect, followed by a disconnect?<u></u><u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">In current implement, the connect,</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
function glusterd_rpc_create called in glusterd_brick_connect not specify option "transport.socket.</span><span lang="EN-US">
</span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">ignore-enoent ".<u></u><u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">So in connect function is
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">socket_connect</span>”<span lang="EN-US"> in socket.c
</span></span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">will use connect instead of connect_loop, in most case it will return ret with -1 and errno=ENOENT, and it will treat as SUCCESS!(attach
the code below)<u></u><u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And epoll mechanism will introduce a EPOLLHUP event with a</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
not really connected socket! So it always trigger a EPOLLHUP handler which will disconnect the socket, and set the start_triggerred flag to faluse for the brick.<u></u><u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">This disconnect issue is 100% reproduced, You could check the glusterd.log to confirm.<u></u><u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------<u></u><u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> if (ign_enoent) {<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> ret = connect_loop (priv->sock,<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> <wbr> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> SA (&this->peerinfo.sockaddr),<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> <wbr> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> this->peerinfo.sockaddr_len);<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> <wbr> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">} else {<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> ret = connect (priv->sock,<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> <wbr> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> SA (&this->peerinfo.sockaddr),<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> <wbr> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> this->peerinfo.sockaddr_len);<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> }<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><i><span style="font-size:10.5pt;font-family:DengXian">……<span lang="EN-US"><u></u><u></u></span></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">if (ret == -1 && ((errno != EINPROGRESS) && (errno != ENOENT))) {<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
</span></i><i><span style="font-size:10.5pt;font-family:DengXian">……………<span lang="EN-US">.<u></u><u></u></span></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> }<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> else {<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> /* reset connect_failed so that any previous attempts<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> state is not carried forward */<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> priv->connect_failed = 0;<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> ret = 0;<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> }<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------<wbr>------------------------------<wbr>------------------------------<wbr>--------------<u></u><u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt">
<u></u><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><span>2)<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Why glusterfsd start twice ?<u></u><u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">On cluster SN, the brick started with the event ACC event which triggered by friend storage node, it will finally called
in function glusterd_friend_sm as the following code <u></u><u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------<wbr>------------------------------<wbr>--------------------<u></u><u></u></span></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> synclock_unlock (&priv->big_lock);<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> glusterd_launch_synctask (glusterd_spawn_daemons, NULL);<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> synclock_lock (&priv->big_lock);<u></u><u></u></span></i></p>
<p class="m_-7482893876161546093MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> glusterd_do_quorum_action ();<u></u><u></u></span></i></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">------------------------------<wbr>------------------------------<wbr>---------------<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">From the above code, the brick start maybe trigger twice, one is glusterd_spawn_saemons, another is glusterd_do_quorum_action.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And it is protected by the flag
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">brickinfo->start_triggered</span>”<span lang="EN-US">, which used to prevent one brick started twice.<u></u><u></u></span></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">BUT, when the disconnect issue is always be there as analyzed as the above, it will be occasionally destroy the protect mechanism,
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">which is to say, before the second
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">brick start</span>”<span lang="EN-US"> begin, the first
</span>“<span lang="EN-US">brick start</span>”<span lang="EN-US"> on same brick has EPOLLHUP event arrived, and the flag
</span>“<span lang="EN-US">brickinfo->start_triggered</span>”<span lang="EN-US"> has change to false , then the second
</span>“<span lang="EN-US">brick start</span>”<span lang="EN-US"> will treat it as a fresh brick process, which we has find in the log of
</span>“<span lang="EN-US">glusterd.log</span>”<span lang="EN-US">, abstract one of example as the below:<u></u><u></u></span></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> Line 632: [2018-04-15 18:06:00.277399] I [glusterd-utils.c:5928:<wbr>glusterd_brick_start] 0-management: starting
a fresh brick process for brick /mnt/bricks/log/brick<u></u><u></u></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.5pt"> </span></i><i><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> Line 661: [2018-04-15 18:06:00.764239] I [glusterd-utils.c:5928:<wbr>glusterd_brick_start] 0-management: starting
a fresh brick process for brick /mnt/bricks/log/brick<u></u><u></u></span></i></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">And when this occasional issue happen, the glustershd service will abnormal finally.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">So I suppose it is a defect for current implement, and a suggest change solution is one code change in function
</span><span style="font-size:10.5pt;font-family:DengXian">“<span lang="EN-US">socket_connect</span>”<span lang="EN-US"> in socket.c as the below:<u></u><u></u></span></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">/******<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> ign_enoent = dict_get_str_boolean (this->options,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> "transport.socket.ignore-<wbr>enoent", _gf_false);</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Change to ****/<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> ign_enoent = dict_get_str_boolean (this->options,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"> "transport.socket.ignore-<wbr>enoent", _gf_true);</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">
<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">After test, the disconnect issue will not happen anymore!<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">@all expert from gluster,
</span><span lang="EN-US" style="font-size:10.5pt"> </span><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">could you please review my investigation as the above, and share your comments?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Also, maybe my change will lead to other issue, so could you please give the right solutions?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Thanks & Best Regards<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">George<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Atin Mukherjee [<a href="mailto:amukherj@redhat.com" target="_blank">mailto:amukherj@redhat.com</a>]
<br>
<b>Sent:</b> Thursday, December 14, 2017 12:37 PM<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>><br>
<b>Cc:</b> Milind Changire <<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>>; Gaurav Yadav <<a href="mailto:gyadav@redhat.com" target="_blank">gyadav@redhat.com</a>>; Mohit Agrawal <<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>>; Raghavendra Gowdappa
<<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>; Samikshan Bairagya <<a href="mailto:sbairagy@redhat.com" target="_blank">sbairagy@redhat.com</a>><br>
<b>Subject:</b> Re: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"><u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">That would be a hack. I think we need to first figure out on a successful brick connect why do we receive a connect, followed by a disconnect event and then another connect event. That's where the problem exists. I'm
waiting to hear for inputs from our RPC experts on this.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Thu, Dec 14, 2017 at 8:20 AM, Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>> wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Hi
</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Mukherjee,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">I will try to add some sleep in glusterd_volume_start_<wbr>glusterfs, before it execute glusterd_brick_connect,
to see if this could help to avoid the “unexpected disconnect event”, I think if this works this race condition could be avoided.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">（周琳）</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">MBB SM HETRAN SW3 MATRIX
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Storage
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Zhou,
Cynthia (NSB - CN/Hangzhou) <br>
<b>Sent:</b> Wednesday, December 13, 2017 4:18 PM<br>
<b>To:</b> 'Atin Mukherjee' <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>>; Milind Changire <<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>><br>
<b>Cc:</b> Gaurav Yadav <<a href="mailto:gyadav@redhat.com" target="_blank">gyadav@redhat.com</a>>; Mohit Agrawal <<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>>; Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>;
Samikshan Bairagya <<a href="mailto:sbairagy@redhat.com" target="_blank">sbairagy@redhat.com</a>><br>
<b>Subject:</b> RE: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">@</span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Mukherjee
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">When I check the code , I find glusterd_volume_start_<wbr>glusterfs will call glusterd_brick_connect, is
it possible that the glusterd process try to connect to glusterfsd process too early so that it will receive disconnect event?</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">（周琳）</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">MBB SM HETRAN SW3 MATRIX
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Storage
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Atin
Mukherjee [mailto:<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>]
<br>
<b>Sent:</b> Tuesday, December 12, 2017 9:00 PM<br>
<b>To:</b> Milind Changire <<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>><br>
<b>Cc:</b> Gaurav Yadav <<a href="mailto:gyadav@redhat.com" target="_blank">gyadav@redhat.com</a>>; Mohit Agrawal <<a href="mailto:moagrawa@redhat.com" target="_blank">moagrawa@redhat.com</a>>; Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>;
Samikshan Bairagya <<a href="mailto:sbairagy@redhat.com" target="_blank">sbairagy@redhat.com</a>>; Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>><br>
<b>Subject:</b> Re: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, 12 Dec 2017 at 17:43, Milind Changire <<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>> wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span lang="EN-US">@Cynthia:<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">What is the command(s) used to restart gluster processes ?<u></u><u></u></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">pkill gluster; glusterd is what I did to simulate this.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, Dec 12, 2017 at 3:25 PM, Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">Looks like we have a race here. I can see a brick disconnect (RPC_CLNT_DISCONNECT) event when the brick process was coming up.<u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.488767] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.489020] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/export/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.489450] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.489717] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/log/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.490114] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.490398] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/mstate/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.490785] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">What I can see is every time when I restart all the gluster process (simulating a reboot scenario) I could see respective brick disconnect events being received at glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US"><br>
#0 __glusterd_brick_rpc_notify (rpc=rpc@entry=0x7fffe8008810, mydata=mydata@entry=<wbr>0x7fffe800e0d0,
<br>
event=event@entry=RPC_CLNT_<wbr>DISCONNECT, data=data@entry=0x0) at glusterd-handler.c:6007<br>
#1 0x00007ffff2936169 in glusterd_big_locked_notify (rpc=0x7fffe8008810, mydata=0x7fffe800e0d0, event=RPC_CLNT_DISCONNECT,
<br>
data=0x0, notify_fn=0x7ffff2932440 <__glusterd_brick_rpc_notify>) at glusterd-handler.c:68<br>
#2 0x00007ffff78c0343 in rpc_clnt_handle_disconnect (conn=0x7fffe8008840, clnt=0x7fffe8008810) at rpc-clnt.c:876<br>
#3 rpc_clnt_notify (trans=0x7fffe8008ae0, mydata=0x7fffe8008840, event=RPC_TRANSPORT_<wbr>DISCONNECT, data=<optimized out>)<br>
at rpc-clnt.c:939<br>
#4 0x00007ffff78bca73 in rpc_transport_notify (this=this@entry=<wbr>0x7fffe8008ae0, event=event@entry=RPC_<wbr>TRANSPORT_DISCONNECT,
<br>
data=data@entry=<wbr>0x7fffe8008ae0) at rpc-transport.c:545<br>
#5 0x00007fffefc252bf in socket_event_poll_err (idx=<optimized out>, gen=<optimized out>, this=0x7fffe8008ae0) at socket.c:1210<br>
#6 socket_event_handler (fd=9, idx=<optimized out>, gen=<optimized out>, data=0x7fffe8008ae0, poll_in=0, poll_out=<optimized out>,
<br>
poll_err=16) at socket.c:2488<br>
#7 0x00007ffff7b5097c in event_dispatch_epoll_handler (event=0x7fffee9dce84, event_pool=0x64f040) at event-epoll.c:583<br>
#8 event_dispatch_epoll_worker (data=0x70b820) at event-epoll.c:659<br>
#9 0x00007ffff693f36d in start_thread () from /lib64/libpthread.so.0<br>
#10 0x00007ffff61ecbbf in clone () from /lib64/libc.so.6<u></u><u></u></span></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span lang="EN-US">The question what I have for RPC team is - why do we see a disconnect event in this case? Any pointers, Mohit/Raghu/Milind?<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">The side effect of this disconnect event is what caused this issue. In __glusterd_brick_rpc_notify we set the brickinfo->start_triggered to false. So when two
glusterd_brick_start threads were racing with each other, in between we got a disconnect because of which this flag got reset and we ended up trying to spawn the same brick twice.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, Dec 12, 2017 at 11:05 AM, Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>
wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Ok. When this happened.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">gluster volume info mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Volume Name: mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Type: Replicate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Volume ID: 5fc29728-b5de-4b5f-8f3c-<wbr>9654acda2577</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Status: Started</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Snapshot Count: 0</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Number of Bricks: 1 x 2 = 2</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Transport-type: tcp</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Bricks:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Brick1: sn-0.local:/mnt/bricks/mstate/<wbr>brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Brick2: sn-1.local:/mnt/bricks/mstate/<wbr>brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Options Reconfigured:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">nfs.disable: on</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">transport.address-family: inet</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">cluster.server-quorum-type: none</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">cluster.consistent-metadata: on</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">server.allow-insecure: on</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">network.ping-timeout: 42</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">performance.client-io-threads: off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">cluster.server-quorum-ratio: 51%</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[root@sn-0:/root]</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Also the ps result:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">ps aux | grep gluster</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 380 0.0 0.6 377892 13760 ? Ss 20:07 0:00 python /usr/sbin/glustereventsd --pid-file
/var/run/glustereventsd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 409 0.0 0.4 82964 9148 ? S 20:07 0:00 python /usr/sbin/glustereventsd --pid-file
/var/run/glustereventsd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 811 0.5 1.0 528720 21636 ? Ssl 20:07 0:01 /usr/sbin/glusterd --volfile=/opt/nokia/libexec/<wbr>StorageUtils/etc/glusterd/<wbr>glusterd.vol
-p /run/glusterd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 1088 0.1 0.6 1036648 13996 ? Ssl 20:07 0:00 /usr/sbin/glusterfsd -s sn-0.local
--volfile-id export.sn-0.local.mnt-bricks-<wbr>export-brick -p /var/run/gluster/vols/export/<wbr>sn-0.local-mnt-bricks-export-<wbr>brick.pid -S /var/run/gluster/<wbr>44f098f14fe45b74b61c5a1d98993d<wbr>15.socket --brick-name /mnt/bricks/export/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-export-brick.log
--xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49152 --xlator-option export-server.listen-port=<wbr>49152 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 1153 1.8 0.7 1234036 14664 ? Ssl 20:07 0:06 /usr/sbin/glusterfsd -s sn-0.local
--volfile-id log.sn-0.local.mnt-bricks-log-<wbr>brick -p /var/run/gluster/vols/log/sn-<wbr>0.local-mnt-bricks-log-brick.<wbr>pid -S /var/run/gluster/<wbr>0af76d9541921cccad34e5fd3041a4<wbr>4d.socket --brick-name /mnt/bricks/log/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-log-brick.log
--xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49156 --xlator-option log-server.listen-port=49156 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 1155 0.0 0.5 772996 10580 ? Ssl 20:07 0:00 /usr/sbin/glusterfsd -s sn-0.local
--volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid -S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket --brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log
--xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49154 --xlator-option mstate-server.listen-port=<wbr>49154 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 1167 0.4 0.7 1235512 15488 ? Ssl 20:07 0:01 /usr/sbin/glusterfsd -s sn-0.local
--volfile-id services.sn-0.local.mnt-<wbr>bricks-services-brick -p /var/run/gluster/vols/<wbr>services/sn-0.local-mnt-<wbr>bricks-services-brick.pid -S /var/run/gluster/<wbr>fc16d9fe6fd0695555a5592343da7b<wbr>a3.socket --brick-name /mnt/bricks/services/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-services-brick.log
--xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49155 --xlator-option services-server.listen-port=<wbr>49155 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 1188 0.0 0.5 644916 11896 ? Ssl 20:07 0:00 /usr/sbin/glusterfs -s sn-0.local --volfile-id
gluster/glustershd -p /var/run/gluster/glustershd/<wbr>glustershd.pid -l /var/log/glusterfs/glustershd.<wbr>log -S /var/run/gluster/<wbr>c405d28a3a79d88436a826d75cf715<wbr>3a.socket --xlator-option *replicate*.node-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 2324 0.0 0.6 629784 13932 ? Ssl 20:07 0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local
--volfile-server=sn-1.local --volfile-id=/services /mnt/services</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 2422 0.0 0.5 431036 10604 ? Ssl 20:07 0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local
--volfile-server=sn-1.local --volfile-id=/export /mnt/export</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 2520 0.0 0.5 498712 11024 ? Ssl 20:07 0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local
--volfile-server=sn-1.local --volfile-id=/log /mnt/log</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 2618 0.0 0.5 431036 10476 ? Ssl 20:07 0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local
--volfile-server=sn-1.local --volfile-id=/mstate /mnt/mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 8136 0.0 0.0 8224 784 pts/0 S+ 20:13 0:00 grep gluster</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[root@sn-0:/root]</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">#</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">（周琳）</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">MBB SM HETRAN SW3 MATRIX
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Storage
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Atin
Mukherjee [mailto:</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">amukherj@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">]
<br>
<b>Sent:</b> Tuesday, December 12, 2017 1:32 PM<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <</span><span lang="EN-US"><a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">cynthia.zhou@nokia-sbell.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Cc:</b> Gaurav Yadav <</span><span lang="EN-US"><a href="mailto:gyadav@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">gyadav@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">>;
Samikshan Bairagya <</span><span lang="EN-US"><a href="mailto:sbairagy@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">sbairagy@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Subject:</b> Re: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">I was looking for output of gluster volume info to understand what all volume tunables were turned on.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, Dec 12, 2017 at 10:56 AM, Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>
wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Hi,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">From our test case when this issue reproduced:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">gluster volume status mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Status of volume: mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Gluster process <wbr> TCP Port RDMA Port Online Pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">------------------------------<wbr>------------------------------<wbr>------------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Brick sn-0.local:/mnt/bricks/mstate/<wbr>brick 49157 0 Y 1155
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Brick sn-1.local:/mnt/bricks/mstate/<wbr>brick 49154 0 Y 1201
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Self-heal Daemon on localhost N/A N/A Y 1188
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Self-heal Daemon on sn-2.local N/A N/A Y 27109</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Self-heal Daemon on sn-1.local N/A N/A Y 4836
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Task Status of Volume mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">------------------------------<wbr>------------------------------<wbr>------------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">There are no active volume tasks</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[root@sn-0:/root]</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">#</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">（周琳）</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">MBB SM HETRAN SW3 MATRIX
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Storage
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Atin
Mukherjee [mailto:</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">amukherj@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">]
<br>
<b>Sent:</b> Tuesday, December 12, 2017 12:02 PM<br>
<b>To:</b> Zhou, Cynthia (NSB - CN/Hangzhou) <</span><span lang="EN-US"><a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">cynthia.zhou@nokia-sbell.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Cc:</b> Gaurav Yadav <</span><span lang="EN-US"><a href="mailto:gyadav@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">gyadav@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">>;
Samikshan Bairagya <</span><span lang="EN-US"><a href="mailto:sbairagy@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">sbairagy@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Subject:</b> Re: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Tue, Dec 12, 2017 at 8:34 AM, Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>
wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">By the way , when I check the code I find that in</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">glusterd_op_perform_add_bricks and glusterd_start_volume, glusterd_brick_start is not triggered within
pthread_mutex_lock, do you think we could add pthread_mutex_lock?</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I don't think that's needed as these functions are triggered through CLI command. We get into this race when the brick start is triggered asynchronously from
a restart glusterd code path. Do you have the volume info output when this issue occurred?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">（周琳）</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">MBB SM HETRAN SW3 MATRIX
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Storage
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Zhou,
Cynthia (NSB - CN/Hangzhou) <br>
<b>Sent:</b> Tuesday, December 12, 2017 9:23 AM<br>
<b>To:</b> 'Gaurav Yadav' <</span><span lang="EN-US"><a href="mailto:gyadav@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">gyadav@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">>;
Atin Mukherjee <</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">amukherj@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Cc:</b> Samikshan Bairagya <</span><span lang="EN-US"><a href="mailto:sbairagy@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">sbairagy@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Subject:</b> RE: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">HI,
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Thanks for responding!</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">My testing version has include the correction of bug-1506589, I’ve also checked my gluster source code.
so this is a new issue.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Unfortunately this issue rarely appear, It only appeared once until now. it is found after reboot sn
node.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Enclosed is the content in /var/lib/glusterd and /var/log/glusterfs</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">The time when this issue happened is , only mstate volume has this issue.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[<span style="color:red">2017-12-09 18:07:02.743046</span>] W [glusterfsd.c:1375:cleanup_<wbr>and_exit] (-->/lib64/libpthread.so.0(+<wbr>0x7454)
[0x7f5df4db6454] -->/usr/sbin/glusterfsd(<wbr>glusterfs_sigwaiter+0xdc) [0x40a50c] -->/usr/sbin/glusterfsd(<wbr>cleanup_and_exit+0x77) [0x4089ec] ) 0-: received signum (15), shutting down</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[<span style="color:red">2017-12-09 18:07:34.416169</span>] I [MSGID: 100030] [glusterfsd.c:2524:main]
0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.12.3 (args: /usr/sbin/glusterfsd -s sn-0.local --volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid -S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket
--brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49154 --xlator-option mstate-server.listen-port=<wbr>49154 --xlator-option transport.socket.bind-address=<wbr>sn-0.local)</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:red">[2017-12-09 18:07:34.511598</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">]
I [MSGID: 100030] [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.12.3 (args: /usr/sbin/glusterfsd -s sn-0.local --volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid
-S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket --brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49157 --xlator-option
mstate-server.listen-port=<wbr>49157 --xlator-option transport.socket.bind-address=<wbr>sn-0.local)</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Best regards,<br>
<b>Cynthia </b></span><b><span style="font-size:10.5pt">（周琳）</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">MBB SM HETRAN SW3 MATRIX
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Arial",sans-serif">Storage
<b><br>
</b>Mobile: +86 (0)18657188311</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Gaurav
Yadav [</span><span lang="EN-US"><a href="mailto:gyadav@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">mailto:gyadav@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">]
<br>
<b>Sent:</b> Monday, December 11, 2017 6:58 PM<br>
<b>To:</b> Atin Mukherjee <</span><span lang="EN-US"><a href="mailto:amukherj@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">amukherj@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Cc:</b> Samikshan Bairagya <</span><span lang="EN-US"><a href="mailto:sbairagy@redhat.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">sbairagy@redhat.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">>;
Zhou, Cynthia (NSB - CN/Hangzhou) <</span><span lang="EN-US"><a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">cynthia.zhou@nokia-sbell.com</span></a></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">><br>
<b>Subject:</b> Re: glusterfsd start twice leading to glustershd can not connect with glusterfsd</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
After seeing logs which you have shared here, it seems it's a port mismatch issue.
<br>
We had a similar issue <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1506589" target="_blank">
bug-1506589,</a> "Please confirm that you too are seeing same behavior".<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><br>
If not please share setup details along with.. <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">* files from /var/lib/glusterd/ directory,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">* log files from /var/log/glusterfs<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Thanks<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US">Gaurav<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span lang="EN-US">On Mon, Dec 11, 2017 at 3:16 PM, Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<u></u><u></u></span></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Adding couple of my colleagues to the thread since I will be busy through out this week. If this is easily reproducible then could you share the exact steps?<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span lang="EN-US"> <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Mon, 11 Dec 2017 at 14:56, Zhou, Cynthia (NSB - CN/Hangzhou) <<a href="mailto:cynthia.zhou@nokia-sbell.com" target="_blank">cynthia.zhou@nokia-sbell.com</a>>
wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Hi,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> Good day!</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif">
</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">I see your change
</span><span lang="EN-US" style="font-size:8.5pt;font-family:"Courier New"">I292f1e58d6971e111725e1baea1fe<wbr>98b890b43e2 i</span><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">n glusterfs code.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> However from my test glusterfs 3.12.3 (already contain that modification) still has something wrong,</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">gluster volume status mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Status of volume: mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Gluster process <wbr> TCP Port RDMA Port Online Pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">------------------------------<wbr>------------------------------<wbr>------------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Brick sn-0.local:/mnt/bricks/mstate/<wbr>brick
<b><span style="color:red">49157</span></b> 0 Y 1155 </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Brick sn-1.local:/mnt/bricks/mstate/<wbr>brick 49154 0 Y 1201
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Self-heal Daemon on localhost N/A N/A Y 1188
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Self-heal Daemon on sn-2.local N/A N/A Y 27109</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Self-heal Daemon on sn-1.local N/A N/A Y 4836
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Task Status of Volume mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">------------------------------<wbr>------------------------------<wbr>------------------</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">There are no active volume tasks</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif">
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">But ps shows</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">ps aux | grep gluster</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 380 0.0 0.6 377892 13760 ? Ss 20:07 0:00 python /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 409 0.0 0.4 82964 9148 ? S 20:07 0:00 python /usr/sbin/glustereventsd --pid-file /var/run/glustereventsd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 811 0.5 1.0 528720 21636 ? Ssl 20:07 0:01 /usr/sbin/glusterd --volfile=/opt/nokia/libexec/<wbr>StorageUtils/etc/glusterd/<wbr>glusterd.vol -p /run/glusterd.pid</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 1088 0.1 0.6 1036648 13996 ? Ssl 20:07 0:00 /usr/sbin/glusterfsd -s sn-0.local --volfile-id export.sn-0.local.mnt-bricks-<wbr>export-brick -p /var/run/gluster/vols/export/<wbr>sn-0.local-mnt-bricks-export-<wbr>brick.pid
-S /var/run/gluster/<wbr>44f098f14fe45b74b61c5a1d98993d<wbr>15.socket --brick-name /mnt/bricks/export/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-export-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49152 --xlator-option
export-server.listen-port=<wbr>49152 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 1153 1.8 0.7 1234036 14664 ? Ssl 20:07 0:06 /usr/sbin/glusterfsd -s sn-0.local --volfile-id log.sn-0.local.mnt-bricks-log-<wbr>brick -p /var/run/gluster/vols/log/sn-<wbr>0.local-mnt-bricks-log-brick.<wbr>pid
-S /var/run/gluster/<wbr>0af76d9541921cccad34e5fd3041a4<wbr>4d.socket --brick-name /mnt/bricks/log/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-log-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49156 --xlator-option
log-server.listen-port=49156 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 1155 0.0 0.5 772996 10580 ? Ssl 20:07 0:00 /usr/sbin/glusterfsd -s sn-0.local --volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid
-S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket --brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49154 --xlator-option
mstate-server.listen-port=<b><span style="color:red">4915<wbr>4</span></b> --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 1167 0.4 0.7 1235512 15488 ? Ssl 20:07 0:01 /usr/sbin/glusterfsd -s sn-0.local --volfile-id services.sn-0.local.mnt-<wbr>bricks-services-brick -p /var/run/gluster/vols/<wbr>services/sn-0.local-mnt-<wbr>bricks-services-brick.pid
-S /var/run/gluster/<wbr>fc16d9fe6fd0695555a5592343da7b<wbr>a3.socket --brick-name /mnt/bricks/services/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-services-brick.log --xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port 49155 --xlator-option
services-server.listen-port=<wbr>49155 --xlator-option transport.socket.bind-address=<wbr>sn-0.local</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 1188 0.0 0.5 644916 11896 ? Ssl 20:07 0:00 /usr/sbin/glusterfs -s sn-0.local --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/<wbr>glustershd.pid -l
/var/log/glusterfs/glustershd.<wbr>log -S /var/run/gluster/<wbr>c405d28a3a79d88436a826d75cf715<wbr>3a.socket --xlator-option *replicate*.node-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 2324 0.0 0.6 629784 13932 ? Ssl 20:07 0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local --volfile-server=sn-1.local --volfile-id=/services /mnt/services</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 2422 0.0 0.5 431036 10604 ? Ssl 20:07 0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local --volfile-server=sn-1.local --volfile-id=/export /mnt/export</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 2520 0.0 0.5 498712 11024 ? Ssl 20:07 0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local --volfile-server=sn-1.local --volfile-id=/log /mnt/log</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 2618 0.0 0.5 431036 10476 ? Ssl 20:07 0:00 /usr/sbin/glusterfs --acl --volfile-server=sn-0.local --volfile-server=sn-1.local --volfile-id=/mstate /mnt/mstate</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">root 8136 0.0 0.0 8224 784 pts/0 S+ 20:13 0:00 grep gluster</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[root@sn-0:/root]</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<b><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">So glustershd can not connect with mstate brick on sn-0</span></b><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">gluster volume heal mstate info</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Brick sn-0.local:/mnt/bricks/mstate/<wbr>brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<b><span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:red">Status: Transport endpoint is not connected</span></b><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Number of entries: -</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Brick sn-1.local:/mnt/bricks/mstate/<wbr>brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Status: Connected</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Number of entries: 0</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">From sn-0 glusterd log I find:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.366851] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/export/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.368484] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.406488] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/log/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.410384] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:red">[2017-12-09 18:07:34.410530] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.413819] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.413926] C [MSGID: 106003] [glusterd-server-quorum.c:352:<wbr>glusterd_do_volume_quorum_<wbr>action] 0-management: Server quorum regained for volume services. Starting local
bricks.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.414031] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/services/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.417573] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.462163] I [MSGID: 106492] [glusterd-handler.c:2718:__<wbr>glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: e4ea0550-873b-410c-a29e-<wbr>3901c4737cd8</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.470228] I [MSGID: 106502] [glusterd-handler.c:2763:__<wbr>glusterd_handle_friend_update] 0-management: Received my uuid as Friend</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.470432] I [MSGID: 106492] [glusterd-handler.c:2718:__<wbr>glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: feddd74c-c768-4157-b152-<wbr>31d5dd488dcc</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.470493] I [MSGID: 106502] [glusterd-handler.c:2763:__<wbr>glusterd_handle_friend_update] 0-management: Received my uuid as Friend</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.479540] I [MSGID: 106163] [glusterd-handshake.c:1316:__<wbr>glusterd_mgmt_hndsk_versions_<wbr>ack] 0-management: using the op-version 31202</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.488767] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.489020] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/export/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.489450] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.489717] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/log/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.490114] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.490398] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/mstate/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.490785] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.491049] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/<wbr>services/brick has disconnected from
glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.491514] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__<wbr>glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: feddd74c-c768-4157-b152-<wbr>31d5dd488dcc, host: sn-1.local,
port: 0</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.501902] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/log/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.503376] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif;color:red">[2017-12-09 18:07:34.503502] I [glusterd-utils.c:5926:<wbr>glusterd_brick_start] 0-management: starting a fresh brick process for brick /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.506655] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-management: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.507887] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.508188] I [MSGID: 106005] [glusterd-handler.c:6071:__<wbr>glusterd_brick_rpc_notify] 0-management: Brick sn-0.local:/mnt/bricks/log/<wbr>brick has disconnected from glusterd.</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.509479] I [socket.c:2478:socket_event_<wbr>handler] 0-transport: EPOLLERR - disconnecting now</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">……</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.023087] I [glusterd-utils.c:5834:<wbr>glusterd_brick_start] 0-management: discovered already-running brick /mnt/bricks/export/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.023106] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/export/brick on port 49152</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.060419] I [glusterd-utils.c:5834:<wbr>glusterd_brick_start] 0-management: discovered already-running brick /mnt/bricks/log/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.060442] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/log/brick on port 49156</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.094987] I [glusterd-utils.c:5834:<wbr>glusterd_brick_start] 0-management: discovered already-running brick /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.095008] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/mstate/brick on port
<span style="color:red">49157</span></span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.129059] I [glusterd-utils.c:5834:<wbr>glusterd_brick_start] 0-management: discovered already-running brick /mnt/bricks/services/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.129108] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/services/brick on port 49155</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.159408] I [rpc-clnt.c:1044:rpc_clnt_<wbr>connection_init] 0-snapd: setting frame-timeout to 600</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">……</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.919192] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__<wbr>glusterd_friend_update_cbk] 0-management: Received ACC from uuid: e4ea0550-873b-410c-a29e-<wbr>3901c4737cd8</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.920269] I [MSGID: 106492] [glusterd-handler.c:2718:__<wbr>glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: e4ea0550-873b-410c-a29e-<wbr>3901c4737cd8</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.927413] I [MSGID: 106502] [glusterd-handler.c:2763:__<wbr>glusterd_handle_friend_update] 0-management: Received my uuid as Friend</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.949762] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__<wbr>glusterd_friend_update_cbk] 0-management: Received ACC from uuid: feddd74c-c768-4157-b152-<wbr>31d5dd488dcc</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.949817] I [MSGID: 106492] [glusterd-handler.c:2718:__<wbr>glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: feddd74c-c768-4157-b152-<wbr>31d5dd488dcc</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:36.949882] I [MSGID: 106502] [glusterd-handler.c:2763:__<wbr>glusterd_handle_friend_update] 0-management: Received my uuid as Friend</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:37.768326] W [MSGID: 101095] [xlator.c:213:xlator_dynload] 0-xlator: /usr/lib64/glusterfs/3.12.3/<wbr>xlator/nfs/server.so: cannot open shared object file: No such file
or directory</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:37.783811] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/services/brick on port 49155</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:37.783901] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/mstate/brick on port
<span style="color:red">49154</span></span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:37.783966] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/log/brick on port 49156</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:37.784032] I [MSGID: 106143] [glusterd-pmap.c:280:pmap_<wbr>registry_bind] 0-pmap: adding brick /mnt/bricks/export/brick on port 49152</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:37.810305] W [MSGID: 101095] [xlator.c:162:xlator_volopt_<wbr>dynload] 0-xlator: /usr/lib64/glusterfs/3.12.3/<wbr>xlator/nfs/server.so: cannot open shared object file: No such
file or directory</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">And the mstate brick process on sn-0 shows:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:02.743046] W [glusterfsd.c:1375:cleanup_<wbr>and_exit] (-->/lib64/libpthread.so.0(+<wbr>0x7454) [0x7f5df4db6454] -->/usr/sbin/glusterfsd(<wbr>glusterfs_sigwaiter+0xdc) [0x40a50c]
-->/usr/sbin/glusterfsd(<wbr>cleanup_and_exit+0x77) [0x4089ec] ) 0-: received signum (15), shutting down</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.416169] I [MSGID: 100030] [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.12.3 (args: /usr/sbin/glusterfsd -s sn-0.local
--volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid -S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket --brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log
--xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port
<span style="color:red">49154</span> --xlator-option mstate-server.listen-port=<wbr>49154 --xlator-option transport.socket.bind-address=<wbr>sn-0.local)</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.511598] I [MSGID: 100030] [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.12.3 (args: /usr/sbin/glusterfsd -s sn-0.local
--volfile-id mstate.sn-0.local.mnt-bricks-<wbr>mstate-brick -p /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid -S /var/run/gluster/<wbr>79d00394b1085adcf7dc1fd4c57f71<wbr>bc.socket --brick-name /mnt/bricks/mstate/brick -l /var/log/glusterfs/bricks/mnt-<wbr>bricks-mstate-brick.log
--xlator-option *-posix.glusterd-uuid=<wbr>f830d809-c935-451f-853a-<wbr>1364f76a4160 --brick-port
<span style="color:red">49157</span> --xlator-option mstate-server.listen-port=<wbr>49157 --xlator-option transport.socket.bind-address=<wbr>sn-0.local)</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.542268] E [MSGID: 100018] [glusterfsd.c:2109:glusterfs_<wbr>pidfile_update] 0-glusterfsd: pidfile /var/run/gluster/vols/mstate/<wbr>sn-0.local-mnt-bricks-mstate-<wbr>brick.pid
lock failed [Resource temporarily unavailable]</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.549242] W [MSGID: 101002] [options.c:995:xl_opt_<wbr>validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing
with correction</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:34.550119] I [MSGID: 101190] [event-epoll.c:613:event_<wbr>dispatch_epoll_worker] 0-epoll: Started thread with index 1</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:37.690037] I [rpcsvc.c:2322:rpcsvc_set_<wbr>outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">[2017-12-09 18:07:37.690125] W [MSGID: 101002] [options.c:995:xl_opt_<wbr>validate] 0-mstate-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port'<wbr>, continuing
with correction</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Times New Roman",serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">……</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> </span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">Final graph:</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">+-----------------------------<wbr>------------------------------<wbr>-------------------+</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> 1: volume mstate-posix</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> 2: type storage/posix</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> 3: option glusterd-uuid f830d809-c935-451f-853a-<wbr>1364f76a4160</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> 4: option directory /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> 5: option volume-id 5fc29728-b5de-4b5f-8f3c-<wbr>9654acda2577</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> 6: option shared-brick-count 1</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> 7: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> 8:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif"> 9: volume mstate-trash</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">10: type features/trash</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">11: option trash-dir .trashcan</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">12: option brick-path /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">13: option trash-internal-op off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">14: subvolumes mstate-posix</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">15: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">16:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">17: volume mstate-changetimerecorder</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">18: type features/changetimerecorder</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">19: option db-type sqlite3</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">20: option hot-brick off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">21: option db-name brick.db</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">22: option db-path /mnt/bricks/mstate/brick/.<wbr>glusterfs/</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">23: option record-exit off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">24: option ctr_link_consistency off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">25: option ctr_lookupheal_link_timeout 300</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">26: option ctr_lookupheal_inode_timeout 300</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">27: option record-entry on</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">28: option ctr-enabled off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">29: option record-counters off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">30: option ctr-record-metadata-heat off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">31: option sql-db-cachesize 12500</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">32: option sql-db-wal-autocheckpoint 25000</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">33: subvolumes mstate-trash</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">34: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">35:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">36: volume mstate-changelog</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">37: type features/changelog</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">38: option changelog-brick /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">39: option changelog-dir /mnt/bricks/mstate/brick/.<wbr>glusterfs/changelogs</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">40: option changelog-barrier-timeout 120</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">41: subvolumes mstate-changetimerecorder</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">42: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">43:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">44: volume mstate-bitrot-stub</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">45: type features/bitrot-stub</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">46: option export /mnt/bricks/mstate/brick</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">47: option bitrot disable</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">48: subvolumes mstate-changelog</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">49: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">50:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">51: volume mstate-access-control</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">52: type features/access-control</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">53: subvolumes mstate-bitrot-stub</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">54: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">55:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">56: volume mstate-locks</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">57: type features/locks</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">58: subvolumes mstate-access-control</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">59: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">60:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">61: volume mstate-worm</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">62: type features/worm</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">63: option worm off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">64: option worm-file-level off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">65: subvolumes mstate-locks</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">66: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">67:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">68: volume mstate-read-only</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">69: type features/read-only</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">70: option read-only off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">71: subvolumes mstate-worm</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">72: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">73:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">74: volume mstate-leases</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">75: type features/leases</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">76: option leases off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">77: subvolumes mstate-read-only</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">78: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">79:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">80: volume mstate-upcall</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">81: type features/upcall</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">82: option cache-invalidation off</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">83: subvolumes mstate-leases</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">84: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">85:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">86: volume mstate-io-threads</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">87: type performance/io-threads</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">88: subvolumes mstate-upcall</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">89: end-volume</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">90:
</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">91: volume mstate-selinux</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">92: type features/selinux</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">93: option selinux on</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">94: subvolumes mstate-io-threads</span><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal" style="text-align:justify;text-justify:inter-ideograph">
<span lang="EN-US" style="font-size:10.5pt;font-family:"Calibri",sans-serif">95: end-volume<</span><span class="HOEnZb"><font color="#888888"><span lang="EN-US"><u></u><u></u></span></font></span></p><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
</font></span></div><span class="HOEnZb"><font color="#888888">
<div>
<p class="MsoNormal"><span lang="EN-US">--
<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">- Atin (atinm)<u></u><u></u></span></p>
</div>
</font></span></div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
</div>
</div>
</blockquote></div><br></div>