<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;
color:black;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Thanks Rafi for workaround.<o:p></o:p></span></p>
<p><b>>> To find the root cause we need to get logs for the first failure of volume start or volume stop .<o:p></o:p></b></p>
<p class="MsoNormal"><span style="color:#1F497D">Below, exact steps to re-produce the issue and attached log file contents from /varlog/gluster folder(parsed)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><u><span style="color:#1F497D">STEPS to re-produce the issue<o:p></o:p></span></u></b></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">root@hostname:~# gluster volume create home-folder transport tcp,rdma storageN1:/gluster/disk1/home-folder storageN2:/gluster/disk1/home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">volume create: home-folder: success: please start the volume to access data<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">root@hostname:~# <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">root@hostname:~# gluster volume info home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Volume Name: home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Type: Distribute<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Volume ID: 09abd02a-b760-459f-afde-95b374eafc53<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Status: Created<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Snapshot Count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Number of Bricks: 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Transport-type: tcp,rdma<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Bricks:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick1: storageN1:/gluster/disk1/home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick2: storageN2:/gluster/disk1/home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Options Reconfigured:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">performance.readdir-ahead: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">nfs.disable: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">root@hostname:~# gluster volume status home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Volume home-folder is not started<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">root@hostname:~# gluster volume start home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">volume start: home-folder: <b>failed: Commit failed on localhost</b>. Please check log file for details.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">root@hostname:~# gluster volume status home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Volume home-folder is not started<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">root@hostname:~# gluster volume start home-folder force<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">volume start: home-folder: success<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">root@hostname:~# gluster volume status home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Status of volume: home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Gluster process TCP Port RDMA Port Online Pid<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick storageN1:/gluster/disk1/home-folder N/A N/A Y 17737<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick storageN2:/gluster/disk1/home-folder N/A N/A N N/A
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Task Status of Volume home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">------------------------------------------------------------------------------<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">There are no active volume tasks<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">root@hostname:~# gluster volume info home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Volume Name: home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Type: Distribute<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Volume ID: 09abd02a-b760-459f-afde-95b374eafc53<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Status: Started<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Snapshot Count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Number of Bricks: 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Transport-type: tcp,rdma<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Bricks:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick1: storageN1:/gluster/disk1/home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Brick2: storageN2:/gluster/disk1/home-folder<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Options Reconfigured:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">performance.readdir-ahead: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">nfs.disable: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">root@hostname:~#<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">--<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Deepak<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext"> Mohammed Rafi K C [mailto:rkavunga@redhat.com]
<br>
<b>Sent:</b> Friday, February 24, 2017 1:08 AM<br>
<b>To:</b> Deepak Naidu; gluster-users@gluster.org<br>
<b>Subject:</b> Re: [Gluster-users] volume start: data0: failed: Commit failed on localhost.<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>It looks like it is ended up in split brain kind of situation. To find the root cause we need to get logs for the first failure of volume start or volume stop .<o:p></o:p></p>
<p>Or to work around it, you can do a volume start force.<o:p></o:p></p>
<p><o:p> </o:p></p>
<p>Regards<o:p></o:p></p>
<p>Rafi KC<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 02/24/2017 01:36 PM, Deepak Naidu wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">I keep on getting this error when my config.transport is set to both tcp,rdma. The volume doesn’t start. I get the below error during volume start.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">To get around this, I end up delete the volume, then configure either only rdma or tcp. May be I am missing something, just trying to get the volume up.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">root@hostname:~# gluster volume start data0<o:p></o:p></p>
<p class="MsoNormal">volume start: data0: failed: Commit failed on localhost. Please check log file for details.<o:p></o:p></p>
<p class="MsoNormal">root@hostname:~#<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">root@ hostname:~# gluster volume status data0<o:p></o:p></p>
<p class="MsoNormal">Staging failed on storageN2. Error: Volume data0 is not started<o:p></o:p></p>
<p class="MsoNormal">root@ hostname:~<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">=============<o:p></o:p></p>
<p class="MsoNormal">[2017-02-24 08:00:29.923516] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data0<o:p></o:p></p>
<p class="MsoNormal">[2017-02-24 08:00:29.926140] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on storageN2. Error: Volume data0 is not started<o:p></o:p></p>
<p class="MsoNormal">[2017-02-24 08:00:33.770505] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume data0<o:p></o:p></p>
<p class="MsoNormal">[2017-02-24 08:00:33.772824] E [MSGID: 106153] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on storageN2. Error: Volume data0 is not started<o:p></o:p></p>
<p class="MsoNormal">=============<o:p></o:p></p>
<p class="MsoNormal">[2017-02-24 08:01:36.305165] E [MSGID: 106537] [glusterd-volume-ops.c:1660:glusterd_op_stage_start_volume] 0-management: Volume data0 already started<o:p></o:p></p>
<p class="MsoNormal">[2017-02-24 08:01:36.305191] W [MSGID: 106122] [glusterd-mgmt.c:198:gd_mgmt_v3_pre_validate_fn] 0-management: Volume start prevalidation failed.<o:p></o:p></p>
<p class="MsoNormal">[2017-02-24 08:01:36.305198] E [MSGID: 106122] [glusterd-mgmt.c:884:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Start on local node<o:p></o:p></p>
<p class="MsoNormal">[2017-02-24 08:01:36.305205] E [MSGID: 106122] [glusterd-mgmt.c:2009:glusterd_mgmt_v3_initiate_all_phases] 0-management: Pre Validation Failed<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">--<o:p></o:p></p>
<p class="MsoNormal">Deepak<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">
<hr size="2" width="100%" align="center">
</span></div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
<o:p></o:p></span></p>
</div>
<div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:12.0pt;font-family:"Times New Roman","serif"">
<hr size="2" width="100%" align="center">
</span></div>
</div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><br>
<br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>Gluster-users mailing list<o:p></o:p></pre>
<pre><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></pre>
<pre><a href="http://lists.gluster.org/mailman/listinfo/gluster-users">http://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman","serif""><o:p> </o:p></span></p>
</div>
</body>
</html>