<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.gmail-m6625482331490640972gmail-m-5508754583328725641p1, li.gmail-m6625482331490640972gmail-m-5508754583328725641p1, div.gmail-m6625482331490640972gmail-m-5508754583328725641p1
{mso-style-name:gmail-m_6625482331490640972gmail-m_-5508754583328725641p1;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmail-m6625482331490640972gmail-m-5508754583328725641s1
{mso-style-name:gmail-m_6625482331490640972gmail-m_-5508754583328725641s1;}
p.gmail-m6625482331490640972gmail-m-5508754583328725641p2, li.gmail-m6625482331490640972gmail-m-5508754583328725641p2, div.gmail-m6625482331490640972gmail-m-5508754583328725641p2
{mso-style-name:gmail-m_6625482331490640972gmail-m_-5508754583328725641p2;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space
{mso-style-name:gmail-m_6625482331490640972gmail-m_-5508754583328725641apple-converted-space;}
p.gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1, li.gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1, div.gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1
{mso-style-name:gmail-m_6625482331490640972gmail-m_-5508754583328725641m-2110851390084808688p1;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688s1
{mso-style-name:gmail-m_6625482331490640972gmail-m_-5508754583328725641m-2110851390084808688s1;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">OK, log files attached. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Boris<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Karthik Subrahmanya <ksubrahm@redhat.com><br>
<b>Date: </b>Tuesday, April 16, 2019 at 2:52 AM<br>
<b>To: </b>Atin Mukherjee <atin.mukherjee83@gmail.com>, Boris Goldowsky <bgoldowsky@cast.org><br>
<b>Cc: </b>Gluster-users <gluster-users@gluster.org><br>
<b>Subject: </b>Re: [Gluster-users] Volume stuck unable to add a brick<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Mon, Apr 15, 2019 at 9:43 PM Atin Mukherjee <<a href="mailto:atin.mukherjee83@gmail.com">atin.mukherjee83@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal"><a href="mailto:ksubrahm@redhat.com" target="_blank">+Karthik Subrahmanya</a> <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Didn't we we fix this problem recently? Failed to set extended attribute indicates that temp mount is failing and we don't have quorum number of bricks up.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">We had two fixes which handles two kind of add-brick scenarios.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[1] Fails add-brick when increasing the replica count if any of the brick is down to avoid data loss. This can be overridden by using the force option.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2] Allow add-brick to set the extended attributes by the temp mount if the volume is already mounted (has clients).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">They are in version 3.12.2 so, patch [1] is present there. But since they are using the force option it should not have any problem even if they have any brick down. The error message they are getting is also different, so it is not because
of any brick being down I guess.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Patch [2] is not present in 3.12.2 and it is not the conversion from plain distribute to replicate volume. So the scenario is different here.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">It seems like they are hitting some other issue.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">@Boris,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Can you attach the add-brick's temp mount log. The file name should look something like "dockervols-add-brick-mount.log". Can you also provide all the brick logs of that volume during that time.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">[1] <a href="https://review.gluster.org/#/c/glusterfs/+/16330/">https://review.gluster.org/#/c/glusterfs/+/16330/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2] <a href="https://review.gluster.org/#/c/glusterfs/+/21791/">https://review.gluster.org/#/c/glusterfs/+/21791/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Karthik<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Boris - What's the gluster version are you using?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Mon, Apr 15, 2019 at 7:35 PM Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org" target="_blank">bgoldowsky@cast.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Atin, thank you for the reply. Here are all of those pieces of information:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver9 ~]$ gluster --version</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">glusterfs 3.12.2</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">(same on all nodes)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver9 ~]$ sudo gluster peer status</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Peers: 3</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p2"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Hostname:
<a href="http://webserver11.cast.org" target="_blank">webserver11.cast.org</a></span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Uuid: c2b147fd-cab4-4859-9922-db5730f8549d</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">State: Peer in Cluster (Connected)</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p2"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Hostname:
<a href="http://webserver1.cast.org" target="_blank">webserver1.cast.org</a></span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Uuid: 4b918f65-2c9d-478e-8648-81d1d6526d4c</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">State: Peer in Cluster (Connected)</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Other names:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">192.168.200.131</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">webserver1</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p2"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Hostname:
<a href="http://webserver8.cast.org" target="_blank">webserver8.cast.org</a></span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Uuid: be2f568b-61c5-4016-9264-083e4e6453a2</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">State: Peer in Cluster (Connected)</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Other names:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">webserver8</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver1 ~]$ sudo gluster v info</span><span class="gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space"> </span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p2"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space"> </span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver8 ~]$ sudo gluster v info</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p2"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space"> </span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver9 ~]$ sudo gluster v info</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p2"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space"> </span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver11 ~]$ sudo gluster v info</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p2"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space"> </span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver9 ~]$ sudo gluster volume add-brick dockervols replica 4 webserver8:/data/gluster/dockervols force</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">volume add-brick: failed: Commit failed on
<a href="http://webserver8.cast.org" target="_blank">webserver8.cast.org</a>. Please check log file for details.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Webserver8 glusterd.log:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 13:55:42.338197] I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol
req" repeated 2 times between [2019-04-15 13:55:42.338197] and [2019-04-15 13:55:42.341618]</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.445011] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x3a215)
[0x7fe697764215] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe3e9d) [0x7fe69780de9d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fe6a2d16ea5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh
--volname=dockervols --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.445148] I [MSGID: 106578] [glusterd-brick-ops.c:1354:glusterd_op_perform_add_bricks] 0-management:
replica-count is set 4</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.445184] I [MSGID: 106578] [glusterd-brick-ops.c:1364:glusterd_op_perform_add_bricks] 0-management:
type is set 0, need to change it</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.672347] E [MSGID: 106054] [glusterd-utils.c:13863:glusterd_handle_replicate_brick_ops] 0-management:
Failed to set extended attribute trusted.add-brick : Transport endpoint is not connected [Transport endpoint is not connected]</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.693491] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of /tmp/mntmvdFGq
[Transport endpoint is not connected]</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.693597] E [MSGID: 106074] [glusterd-brick-ops.c:2590:glusterd_op_add_brick] 0-glusterd: Unable to add
bricks</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.693637] E [MSGID: 106123] [glusterd-mgmt.c:312:gd_mgmt_v3_commit_fn] 0-management: Add-brick commit
failed.</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.693667] E [MSGID: 106123] [glusterd-mgmt-handler.c:616:glusterd_handle_commit_fn] 0-management: commit
failed on operation Add brick</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Webserver11 log file:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 13:56:29.563270] I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol
req" repeated 2 times between [2019-04-15 13:56:29.563270] and [2019-04-15 13:56:29.566209]</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:33.996866] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x3a215)
[0x7f36de924215] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe3e9d) [0x7f36de9cde9d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f36e9ed6ea5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh
--volname=dockervols --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:33.996979] I [MSGID: 106578] [glusterd-brick-ops.c:1354:glusterd_op_perform_add_bricks] 0-management:
replica-count is set 4</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:33.997004] I [MSGID: 106578] [glusterd-brick-ops.c:1364:glusterd_op_perform_add_bricks] 0-management:
type is set 0, need to change it</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:34.013789] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: nfs already stopped</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:34.013849] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: nfs service is
stopped</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:34.017535] I [MSGID: 106568] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: Stopping glustershd
daemon running in pid: 6087</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.018783] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: glustershd service
is stopped</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.018952] I [MSGID: 106567] [glusterd-svc-mgmt.c:211:glusterd_svc_start] 0-management: Starting glustershd
service</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.028306] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: bitd already stopped</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.028408] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: bitd service is
stopped</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.028601] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: scrub already
stopped</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641p1"><span class="gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.028645] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: scrub service is
stopped</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thank you for taking a look!<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Boris<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Atin Mukherjee <<a href="mailto:atin.mukherjee83@gmail.com" target="_blank">atin.mukherjee83@gmail.com</a>><br>
<b>Date: </b>Friday, April 12, 2019 at 1:10 PM<br>
<b>To: </b>Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org" target="_blank">bgoldowsky@cast.org</a>><br>
<b>Cc: </b>Gluster-users <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
<b>Subject: </b>Re: [Gluster-users] Volume stuck unable to add a brick</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, 12 Apr 2019 at 22:32, Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org" target="_blank">bgoldowsky@cast.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’ve got a replicated volume with three bricks (“1x3=3”), the idea is to have a common set of files that are locally available on all the machines (Scientific Linux 7, which is
essentially CentOS 7) in a cluster.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I tried to add on a fourth machine, so used a command like this:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1">
<span class="gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688s1">sudo gluster volume add-brick dockervols replica 4 webserver8:/data/gluster/dockervols force</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">but the result is:<o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1">
<span class="gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688s1">volume add-brick: failed: Commit failed on webserver1. Please check log file for details.</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1">
<span class="gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688s1">Commit failed on webserver8. Please check log file for details.</span><o:p></o:p></p>
<p class="gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1">
<span class="gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688s1">Commit failed on webserver11. Please check log file for details.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: removing the new brick (this also fails) and trying again.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: checking the logs. The log files are not enlightening to me – I don’t know what’s normal and what’s not.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">From webserver8 & webserver11 could you attach glusterd log files?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Also please share following:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- gluster version? (gluster —version)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- Output of ‘gluster peer status’<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- Output of ‘gluster v info’ from all 4 nodes.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: deleting the brick directory from previous attempt, so that it’s not in the way.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: restarting gluster services<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: rebooting<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: setting up a new volume, replicated to all four machines. This works, so I’m assuming it’s not a networking issue. But still fails with this existing volume that has the
critical data in it.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Running out of ideas. Any suggestions? Thank you!<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Boris<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</blockquote>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--Atin<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>