<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1, li.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1, div.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1
{mso-style-name:gmail-m_1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1
{mso-style-name:gmail-m_1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1;}
p.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p2, li.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p2, div.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p2
{mso-style-name:gmail-m_1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p2;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space
{mso-style-name:gmail-m_1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space;}
p.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1, li.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1, div.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1
{mso-style-name:gmail-m_1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688s1
{mso-style-name:gmail-m_1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688s1;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">That worked! Thank you SO much!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Boris<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Karthik Subrahmanya <ksubrahm@redhat.com><br>
<b>Date: </b>Tuesday, April 16, 2019 at 8:20 AM<br>
<b>To: </b>Boris Goldowsky <bgoldowsky@cast.org><br>
<b>Cc: </b>Atin Mukherjee <atin.mukherjee83@gmail.com>, Gluster-users <gluster-users@gluster.org><br>
<b>Subject: </b>Re: [Gluster-users] Volume stuck unable to add a brick<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Hi Boris, <o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thank you for providing the logs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">The problem here is because of the "auth.allow: 127.0.0.1" setting on the volume.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">When you try to add a new brick to the volume internally replication module will try to set some metadata on the existing bricks to mark pending heal on the new brick, by creating a temporary mount. Because of the auth.allow setting that
mount gets permission errors as seen in the below logs, leading to add-brick failure.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">From data-gluster-dockervols.log-webserver9 :<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">[2019-04-15 14:00:34.226838] I [addr.c:55:compare_addr_and_update] 0-/data/gluster/dockervols: allowed = "127.0.0.1", received addr = "192.168.200.147"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2019-04-15 14:00:34.226895] E [MSGID: 115004] [authenticate.c:224:gf_authenticate] 0-auth: no authentication module is interested in accepting remote-client (null)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2019-04-15 14:00:34.227129] E [MSGID: 115001] [server-handshake.c:848:server_setvolume] 0-dockervols-server: Cannot authenticate client from webserver8.cast.org-55674-2019/04/15-14:00:20:495333-dockervols-client-2-0-0 3.12.2 [Permission
denied]<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">From dockervols-add-brick-mount.log :<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">[2019-04-15 14:00:20.672033] W [MSGID: 114043] [client-handshake.c:1109:client_setvolume_cbk] 0-dockervols-client-2: failed to set the volume [Permission denied]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2019-04-15 14:00:20.672102] W [MSGID: 114007] [client-handshake.c:1138:client_setvolume_cbk] 0-dockervols-client-2: failed to get 'process-uuid' from reply dict [Invalid argument]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2019-04-15 14:00:20.672129] E [MSGID: 114044] [client-handshake.c:1144:client_setvolume_cbk] 0-dockervols-client-2: SETVOLUME on remote-host failed: Authentication failed [Permission denied]<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">[2019-04-15 14:00:20.672151] I [MSGID: 114049] [client-handshake.c:1258:client_setvolume_cbk] 0-dockervols-client-2: sending AUTH_FAILED event<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This is a known issue and we are planning to fix this. For the time being we have a workaround for this.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- Before you try adding the brick set the auth.allow option to default i.e., "*" or you can do this by running "gluster v reset <volname> auth.allow"<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- Add the brick<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- After it succeeds set back the auth.allow option to the previous value.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Karthik<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, Apr 16, 2019 at 5:20 PM Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org">bgoldowsky@cast.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">OK, log files attached.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Boris<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>><br>
<b>Date: </b>Tuesday, April 16, 2019 at 2:52 AM<br>
<b>To: </b>Atin Mukherjee <<a href="mailto:atin.mukherjee83@gmail.com" target="_blank">atin.mukherjee83@gmail.com</a>>, Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org" target="_blank">bgoldowsky@cast.org</a>><br>
<b>Cc: </b>Gluster-users <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
<b>Subject: </b>Re: [Gluster-users] Volume stuck unable to add a brick</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Mon, Apr 15, 2019 at 9:43 PM Atin Mukherjee <<a href="mailto:atin.mukherjee83@gmail.com" target="_blank">atin.mukherjee83@gmail.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a href="mailto:ksubrahm@redhat.com" target="_blank">+Karthik Subrahmanya</a> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Didn't we we fix this problem recently? Failed to set extended attribute indicates that temp mount is failing and we don't have quorum number of bricks up.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We had two fixes which handles two kind of add-brick scenarios.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[1] Fails add-brick when increasing the replica count if any of the brick is down to avoid data loss. This can be overridden by using the force option.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2] Allow add-brick to set the extended attributes by the temp mount if the volume is already mounted (has clients).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">They are in version 3.12.2 so, patch [1] is present there. But since they are using the force option it should not have any problem even if they have any brick down. The error message
they are getting is also different, so it is not because of any brick being down I guess.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Patch [2] is not present in 3.12.2 and it is not the conversion from plain distribute to replicate volume. So the scenario is different here.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">It seems like they are hitting some other issue.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">@Boris,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Can you attach the add-brick's temp mount log. The file name should look something like "dockervols-add-brick-mount.log". Can you also provide all the brick logs of that volume
during that time.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[1] <a href="https://review.gluster.org/#/c/glusterfs/+/16330/" target="_blank">https://review.gluster.org/#/c/glusterfs/+/16330/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[2] <a href="https://review.gluster.org/#/c/glusterfs/+/21791/" target="_blank">https://review.gluster.org/#/c/glusterfs/+/21791/</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Karthik<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Boris - What's the gluster version are you using?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Mon, Apr 15, 2019 at 7:35 PM Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org" target="_blank">bgoldowsky@cast.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Atin, thank you for the reply. Here are all of those pieces of information:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver9 ~]$ gluster --version</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">glusterfs 3.12.2</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">(same on all nodes)<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver9 ~]$ sudo gluster peer status</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Peers: 3</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p2">
<o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Hostname:
<a href="http://webserver11.cast.org" target="_blank">webserver11.cast.org</a></span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Uuid: c2b147fd-cab4-4859-9922-db5730f8549d</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">State: Peer in Cluster (Connected)</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p2">
<o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Hostname:
<a href="http://webserver1.cast.org" target="_blank">webserver1.cast.org</a></span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Uuid: 4b918f65-2c9d-478e-8648-81d1d6526d4c</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">State: Peer in Cluster (Connected)</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Other names:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">192.168.200.131</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">webserver1</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p2">
<o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Hostname:
<a href="http://webserver8.cast.org" target="_blank">webserver8.cast.org</a></span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Uuid: be2f568b-61c5-4016-9264-083e4e6453a2</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">State: Peer in Cluster (Connected)</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Other names:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">webserver8</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver1 ~]$ sudo gluster v info</span><span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space"> </span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p2">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space"> </span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver8 ~]$ sudo gluster v info</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p2">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space"> </span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver9 ~]$ sudo gluster v info</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p2">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space"> </span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver11 ~]$ sudo gluster v info</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p2">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641apple-converted-space"> </span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Type: Replicate</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Status: Started</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Bricks:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">nfs.disable: on</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[bgoldowsky@webserver9 ~]$ sudo gluster volume add-brick dockervols replica 4 webserver8:/data/gluster/dockervols force</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">volume add-brick: failed: Commit failed on
<a href="http://webserver8.cast.org" target="_blank">webserver8.cast.org</a>. Please check log file for details.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Webserver8 glusterd.log:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 13:55:42.338197] I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2019-04-15
13:55:42.338197] and [2019-04-15 13:55:42.341618]</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.445011] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x3a215) [0x7fe697764215] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe3e9d)
[0x7fe69780de9d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fe6a2d16ea5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=dockervols --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.445148] I [MSGID: 106578] [glusterd-brick-ops.c:1354:glusterd_op_perform_add_bricks] 0-management: replica-count is set 4</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.445184] I [MSGID: 106578] [glusterd-brick-ops.c:1364:glusterd_op_perform_add_bricks] 0-management: type is set 0, need to change it</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.672347] E [MSGID: 106054] [glusterd-utils.c:13863:glusterd_handle_replicate_brick_ops] 0-management: Failed to set extended attribute trusted.add-brick
: Transport endpoint is not connected [Transport endpoint is not connected]</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.693491] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of /tmp/mntmvdFGq [Transport endpoint is not connected]</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.693597] E [MSGID: 106074] [glusterd-brick-ops.c:2590:glusterd_op_add_brick] 0-glusterd: Unable to add bricks</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.693637] E [MSGID: 106123] [glusterd-mgmt.c:312:gd_mgmt_v3_commit_fn] 0-management: Add-brick commit failed.</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:20.693667] E [MSGID: 106123] [glusterd-mgmt-handler.c:616:glusterd_handle_commit_fn] 0-management: commit failed on operation Add brick</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Webserver11 log file:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 13:56:29.563270] I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2019-04-15
13:56:29.563270] and [2019-04-15 13:56:29.566209]</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:33.996866] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x3a215) [0x7f36de924215] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe3e9d)
[0x7f36de9cde9d] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f36e9ed6ea5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=dockervols --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:33.996979] I [MSGID: 106578] [glusterd-brick-ops.c:1354:glusterd_op_perform_add_bricks] 0-management: replica-count is set 4</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:33.997004] I [MSGID: 106578] [glusterd-brick-ops.c:1364:glusterd_op_perform_add_bricks] 0-management: type is set 0, need to change it</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:34.013789] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: nfs already stopped</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:34.013849] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: nfs service is stopped</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:34.017535] I [MSGID: 106568] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 6087</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.018783] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: glustershd service is stopped</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.018952] I [MSGID: 106567] [glusterd-svc-mgmt.c:211:glusterd_svc_start] 0-management: Starting glustershd service</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.028306] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: bitd already stopped</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.028408] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: bitd service is stopped</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.028601] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: scrub already stopped</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641s1">[2019-04-15 14:00:35.028645] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: scrub service is stopped</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thank you for taking a look!<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Boris<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Atin Mukherjee <<a href="mailto:atin.mukherjee83@gmail.com" target="_blank">atin.mukherjee83@gmail.com</a>><br>
<b>Date: </b>Friday, April 12, 2019 at 1:10 PM<br>
<b>To: </b>Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org" target="_blank">bgoldowsky@cast.org</a>><br>
<b>Cc: </b>Gluster-users <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
<b>Subject: </b>Re: [Gluster-users] Volume stuck unable to add a brick</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, 12 Apr 2019 at 22:32, Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org" target="_blank">bgoldowsky@cast.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’ve got a replicated volume with three bricks (“1x3=3”), the idea is to have a common set of files that are locally available on all the machines (Scientific Linux 7, which is
essentially CentOS 7) in a cluster.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I tried to add on a fourth machine, so used a command like this:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688s1">sudo gluster volume add-brick dockervols replica 4 webserver8:/data/gluster/dockervols force</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">but the result is:<o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688s1">volume add-brick: failed: Commit failed on webserver1. Please check log file for details.</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688s1">Commit failed on webserver8. Please check log file for details.</span><o:p></o:p></p>
<p class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688p1">
<span class="gmail-m1927314987355169274gmail-m6625482331490640972gmail-m-5508754583328725641m-2110851390084808688s1">Commit failed on webserver11. Please check log file for details.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: removing the new brick (this also fails) and trying again.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: checking the logs. The log files are not enlightening to me – I don’t know what’s normal and what’s not.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">From webserver8 & webserver11 could you attach glusterd log files?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Also please share following:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- gluster version? (gluster —version)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- Output of ‘gluster peer status’<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">- Output of ‘gluster v info’ from all 4 nodes.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: deleting the brick directory from previous attempt, so that it’s not in the way.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: restarting gluster services<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: rebooting<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: setting up a new volume, replicated to all four machines. This works, so I’m assuming it’s not a networking issue. But still fails with this existing volume that has the
critical data in it.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Running out of ideas. Any suggestions? Thank you!<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Boris<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</blockquote>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--
<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">--Atin<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>