<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 24, 2017 at 11:06 PM, Deepak Naidu <span dir="ltr"><<a href="mailto:dnaidu@nvidia.com" target="_blank">dnaidu@nvidia.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="white" lang="EN-US">
<div class="gmail-m_7399988227562190548WordSection1">
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Thanks Rafi for workaround.<u></u><u></u></span></p><span class="gmail-">
<p><b>>> To find the root cause we need to get logs for the first failure of volume start or volume stop .<u></u><u></u></b></p>
</span><p class="MsoNormal"><span style="color:rgb(31,73,125)">Below, exact steps to re-produce the issue and attached log file contents from /varlog/gluster folder(parsed)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><u><span style="color:rgb(31,73,125)">STEPS to re-produce the issue<u></u><u></u></span></u></b></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">root@hostname:~# gluster volume create home-folder transport tcp,rdma storageN1:/gluster/disk1/home-<wbr>folder storageN2:/gluster/disk1/home-<wbr>folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">volume create: home-folder: success: please start the volume to access data<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">root@hostname:~# <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">root@hostname:~# gluster volume info home-folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Volume Name: home-folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Type: Distribute<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Volume ID: 09abd02a-b760-459f-afde-<wbr>95b374eafc53<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Status: Created<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Snapshot Count: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Number of Bricks: 2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Transport-type: tcp,rdma<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Bricks:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Brick1: storageN1:/gluster/disk1/home-<wbr>folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Brick2: storageN2:/gluster/disk1/home-<wbr>folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Options Reconfigured:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">performance.readdir-ahead: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">nfs.disable: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">root@hostname:~# gluster volume status home-folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Volume home-folder is not started<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">root@hostname:~# gluster volume start home-folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">volume start: home-folder: <b>failed: Commit failed on localhost</b>. Please check log file for details.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">root@hostname:~# gluster volume status home-folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Volume home-folder is not started<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">root@hostname:~# gluster volume start home-folder force<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">volume start: home-folder: success<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">root@hostname:~# gluster volume status home-folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Status of volume: home-folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Gluster process <wbr> TCP Port RDMA Port Online Pid<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">------------------------------<wbr>------------------------------<wbr>------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Brick storageN1:/gluster/disk1/home-<wbr>folder N/A N/A Y 17737<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Brick storageN2:/gluster/disk1/home-<wbr>folder N/A N/A N N/A
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"> <u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Task Status of Volume home-folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">------------------------------<wbr>------------------------------<wbr>------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">There are no active volume tasks<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">root@hostname:~# gluster volume info home-folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Volume Name: home-folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Type: Distribute<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Volume ID: 09abd02a-b760-459f-afde-<wbr>95b374eafc53<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Status: Started<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Snapshot Count: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Number of Bricks: 2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Transport-type: tcp,rdma<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Bricks:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Brick1: storageN1:/gluster/disk1/home-<wbr>folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Brick2: storageN2:/gluster/disk1/home-<wbr>folder<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Options Reconfigured:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">performance.readdir-ahead: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">nfs.disable: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">root@hostname:~#<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">--<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Deepak<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-width:1pt medium medium;border-style:solid none none;border-color:rgb(181,196,223) -moz-use-text-color -moz-use-text-color;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10pt;font-family:"tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10pt;font-family:"tahoma","sans-serif";color:windowtext"> Mohammed Rafi K C [mailto:<a href="mailto:rkavunga@redhat.com" target="_blank">rkavunga@redhat.com</a>]
<br>
<b>Sent:</b> Friday, February 24, 2017 1:08 AM<br>
<b>To:</b> Deepak Naidu; <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] volume start: data0: failed: Commit failed on localhost.<u></u><u></u></span></p>
</div>
</div><div><div class="gmail-h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p>It looks like it is ended up in split brain kind of situation. To find the root cause we need to get logs for the first failure of volume start or volume stop .<u></u><u></u></p>
<p>Or to work around it, you can do a volume start force.<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Regards<u></u><u></u></p>
<p>Rafi KC<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On 02/24/2017 01:36 PM, Deepak Naidu wrote:<u></u><u></u></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p class="MsoNormal">I keep on getting this error when my config.transport is set to both tcp,rdma. The volume doesn’t start. I get the below error during volume start.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">To get around this, I end up delete the volume, then configure either only rdma or tcp. May be I am missing something, just trying to get the volume up.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">root@hostname:~# gluster volume start data0<u></u><u></u></p>
<p class="MsoNormal">volume start: data0: failed: Commit failed on localhost. Please check log file for details.<u></u><u></u></p>
<p class="MsoNormal">root@hostname:~#<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">root@ hostname:~# gluster volume status data0<u></u><u></u></p>
<p class="MsoNormal">Staging failed on storageN2. Error: Volume data0 is not started<u></u><u></u></p>
<p class="MsoNormal">root@ hostname:~<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">=============<u></u><u></u></p>
<p class="MsoNormal">[2017-02-24 08:00:29.923516] I [MSGID: 106499] [glusterd-handler.c:4349:__<wbr>glusterd_handle_status_volume] 0-management: Received status volume req for volume data0<u></u><u></u></p>
<p class="MsoNormal">[2017-02-24 08:00:29.926140] E [MSGID: 106153] [glusterd-syncop.c:113:gd_<wbr>collate_errors] 0-glusterd: Staging failed on storageN2. Error: Volume data0 is not started<u></u><u></u></p>
<p class="MsoNormal">[2017-02-24 08:00:33.770505] I [MSGID: 106499] [glusterd-handler.c:4349:__<wbr>glusterd_handle_status_volume] 0-management: Received status volume req for volume data0<u></u><u></u></p>
<p class="MsoNormal">[2017-02-24 08:00:33.772824] E [MSGID: 106153] [glusterd-syncop.c:113:gd_<wbr>collate_errors] 0-glusterd: Staging failed on storageN2. Error: Volume data0 is not started<u></u><u></u></p>
<p class="MsoNormal">=============<u></u><u></u></p>
<p class="MsoNormal">[2017-02-24 08:01:36.305165] E [MSGID: 106537] [glusterd-volume-ops.c:1660:<wbr>glusterd_op_stage_start_<wbr>volume] 0-management: Volume data0 already started<u></u><u></u></p>
<p class="MsoNormal">[2017-02-24 08:01:36.305191] W [MSGID: 106122] [glusterd-mgmt.c:198:gd_mgmt_<wbr>v3_pre_validate_fn] 0-management: Volume start prevalidation failed.<u></u><u></u></p>
<p class="MsoNormal">[2017-02-24 08:01:36.305198] E [MSGID: 106122] [glusterd-mgmt.c:884:glusterd_<wbr>mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Start on local node<u></u><u></u></p>
<p class="MsoNormal">[2017-02-24 08:01:36.305205] E [MSGID: 106122] [glusterd-mgmt.c:2009:<wbr>glusterd_mgmt_v3_initiate_all_<wbr>phases] 0-management: Pre Validation Failed</p></blockquote></div></div></div></div></blockquote><div><br><br></div><div>What you are seeing is one of the side effect of BZ <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1386578">https://bugzilla.redhat.com/show_bug.cgi?id=1386578</a> & <a href="https://review.gluster.org/#/c/15687/">https://review.gluster.org/#/c/15687/</a> has been already posted for review. So in this case, although the volume status shows up that volume is not started, the brick process(es) actually do start. As a workaround please use volume start force one more time.<br><br></div><div>@Rafi - We should try to get this patch in early and make it part of the next release.<br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="white" lang="EN-US"><div class="gmail-m_7399988227562190548WordSection1"><div><div class="gmail-h5"><blockquote style="margin-top:5pt;margin-bottom:5pt"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">--<u></u><u></u></p>
<p class="MsoNormal">Deepak<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div class="MsoNormal" style="text-align:center" align="center"><span style="font-size:12pt;font-family:"times new roman","serif"">
<hr width="100%" size="2" align="center">
</span></div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"times new roman","serif"">This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
<u></u><u></u></span></p>
</div>
<div>
<div class="MsoNormal" style="text-align:center" align="center"><span style="font-size:12pt;font-family:"times new roman","serif"">
<hr width="100%" size="2" align="center">
</span></div>
</div>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"times new roman","serif""><br>
<br>
<br>
<u></u><u></u></span></p>
<pre>______________________________<wbr>_________________<u></u><u></u></pre>
<pre>Gluster-users mailing list<u></u><u></u></pre>
<pre><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><u></u><u></u></pre>
<pre><a href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><u></u><u></u></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:12pt;font-family:"times new roman","serif""><u></u> <u></u></span></p>
</div></div></div>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><br></div><div>~ Atin (atinm)<br></div></div></div></div>
</div></div>