<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.m-2110851390084808688p1, li.m-2110851390084808688p1, div.m-2110851390084808688p1
{mso-style-name:m_-2110851390084808688p1;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.m-2110851390084808688s1
{mso-style-name:m_-2110851390084808688s1;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
margin-bottom:.0001pt;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0in;
margin-bottom:.0001pt;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Atin, thank you for the reply. Here are all of those pieces of information:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">[bgoldowsky@webserver9 ~]$ gluster --version</span><o:p></o:p></p>
<p class="p1"><span class="s1">glusterfs 3.12.2</span><o:p></o:p></p>
<p class="MsoNormal">(same on all nodes)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">[bgoldowsky@webserver9 ~]$ sudo gluster peer status</span><o:p></o:p></p>
<p class="p1"><span class="s1">Number of Peers: 3</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">Hostname: webserver11.cast.org</span><o:p></o:p></p>
<p class="p1"><span class="s1">Uuid: c2b147fd-cab4-4859-9922-db5730f8549d</span><o:p></o:p></p>
<p class="p1"><span class="s1">State: Peer in Cluster (Connected)</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">Hostname: webserver1.cast.org</span><o:p></o:p></p>
<p class="p1"><span class="s1">Uuid: 4b918f65-2c9d-478e-8648-81d1d6526d4c</span><o:p></o:p></p>
<p class="p1"><span class="s1">State: Peer in Cluster (Connected)</span><o:p></o:p></p>
<p class="p1"><span class="s1">Other names:</span><o:p></o:p></p>
<p class="p1"><span class="s1">192.168.200.131</span><o:p></o:p></p>
<p class="p1"><span class="s1">webserver1</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">Hostname: webserver8.cast.org</span><o:p></o:p></p>
<p class="p1"><span class="s1">Uuid: be2f568b-61c5-4016-9264-083e4e6453a2</span><o:p></o:p></p>
<p class="p1"><span class="s1">State: Peer in Cluster (Connected)</span><o:p></o:p></p>
<p class="p1"><span class="s1">Other names:</span><o:p></o:p></p>
<p class="p1"><span class="s1">webserver8</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">[bgoldowsky@webserver1 ~]$ sudo gluster v info</span><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Type: Replicate</span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="p1"><span class="s1">Status: Started</span><o:p></o:p></p>
<p class="p1"><span class="s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="p1"><span class="s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="p1"><span class="s1">Bricks:</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="p1"><span class="s1">nfs.disable: on</span><o:p></o:p></p>
<p class="p1"><span class="s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="p1"><span class="s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="p2"><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Type: Replicate</span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="p1"><span class="s1">Status: Started</span><o:p></o:p></p>
<p class="p1"><span class="s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="p1"><span class="s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="p1"><span class="s1">Bricks:</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="p1"><span class="s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="p1"><span class="s1">nfs.disable: on</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">[bgoldowsky@webserver8 ~]$ sudo gluster v info</span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Type: Replicate</span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="p1"><span class="s1">Status: Started</span><o:p></o:p></p>
<p class="p1"><span class="s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="p1"><span class="s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="p1"><span class="s1">Bricks:</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="p1"><span class="s1">nfs.disable: on</span><o:p></o:p></p>
<p class="p1"><span class="s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="p1"><span class="s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="p2"><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Type: Replicate</span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="p1"><span class="s1">Status: Started</span><o:p></o:p></p>
<p class="p1"><span class="s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="p1"><span class="s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="p1"><span class="s1">Bricks:</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="p1"><span class="s1">nfs.disable: on</span><o:p></o:p></p>
<p class="p1"><span class="s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">[bgoldowsky@webserver9 ~]$ sudo gluster v info</span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Type: Replicate</span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="p1"><span class="s1">Status: Started</span><o:p></o:p></p>
<p class="p1"><span class="s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="p1"><span class="s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="p1"><span class="s1">Bricks:</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="p1"><span class="s1">nfs.disable: on</span><o:p></o:p></p>
<p class="p1"><span class="s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="p1"><span class="s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="p2"><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Type: Replicate</span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="p1"><span class="s1">Status: Started</span><o:p></o:p></p>
<p class="p1"><span class="s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="p1"><span class="s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="p1"><span class="s1">Bricks:</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="p1"><span class="s1">nfs.disable: on</span><o:p></o:p></p>
<p class="p1"><span class="s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">[bgoldowsky@webserver11 ~]$ sudo gluster v info</span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume Name: dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Type: Replicate</span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a</span><o:p></o:p></p>
<p class="p1"><span class="s1">Status: Started</span><o:p></o:p></p>
<p class="p1"><span class="s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">Number of Bricks: 1 x 3 = 3</span><o:p></o:p></p>
<p class="p1"><span class="s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="p1"><span class="s1">Bricks:</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick1: webserver1:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick2: webserver11:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick3: webserver9:/data/gluster/dockervols</span><o:p></o:p></p>
<p class="p1"><span class="s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="p1"><span class="s1">auth.allow: 127.0.0.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="p1"><span class="s1">nfs.disable: on</span><o:p></o:p></p>
<p class="p2"><span class="apple-converted-space"> </span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume Name: testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Type: Replicate</span><o:p></o:p></p>
<p class="p1"><span class="s1">Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332</span><o:p></o:p></p>
<p class="p1"><span class="s1">Status: Started</span><o:p></o:p></p>
<p class="p1"><span class="s1">Snapshot Count: 0</span><o:p></o:p></p>
<p class="p1"><span class="s1">Number of Bricks: 1 x 4 = 4</span><o:p></o:p></p>
<p class="p1"><span class="s1">Transport-type: tcp</span><o:p></o:p></p>
<p class="p1"><span class="s1">Bricks:</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick1: webserver1:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick2: webserver9:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick3: webserver11:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Brick4: webserver8:/data/gluster/testvol</span><o:p></o:p></p>
<p class="p1"><span class="s1">Options Reconfigured:</span><o:p></o:p></p>
<p class="p1"><span class="s1">transport.address-family: inet</span><o:p></o:p></p>
<p class="p1"><span class="s1">nfs.disable: on</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">[bgoldowsky@webserver9 ~]$ sudo gluster volume add-brick dockervols replica 4 webserver8:/data/gluster/dockervols force</span><o:p></o:p></p>
<p class="p1"><span class="s1">volume add-brick: failed: Commit failed on webserver8.cast.org. Please check log file for details.</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Webserver8 glusterd.log:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">[2019-04-15 13:55:42.338197] I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</span><o:p></o:p></p>
<p class="p1"><span class="s1">The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2019-04-15 13:55:42.338197] and [2019-04-15 13:55:42.341618]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:20.445011] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x3a215) [0x7fe697764215] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe3e9d) [0x7fe69780de9d] -->/lib64/libglusterfs.so.0(runner_log+0x115)
[0x7fe6a2d16ea5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=dockervols --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:20.445148] I [MSGID: 106578] [glusterd-brick-ops.c:1354:glusterd_op_perform_add_bricks] 0-management: replica-count is set 4</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:20.445184] I [MSGID: 106578] [glusterd-brick-ops.c:1364:glusterd_op_perform_add_bricks] 0-management: type is set 0, need to change it</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:20.672347] E [MSGID: 106054] [glusterd-utils.c:13863:glusterd_handle_replicate_brick_ops] 0-management: Failed to set extended attribute trusted.add-brick : Transport endpoint is not connected [Transport endpoint
is not connected]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:20.693491] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of /tmp/mntmvdFGq [Transport endpoint is not connected]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:20.693597] E [MSGID: 106074] [glusterd-brick-ops.c:2590:glusterd_op_add_brick] 0-glusterd: Unable to add bricks</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:20.693637] E [MSGID: 106123] [glusterd-mgmt.c:312:gd_mgmt_v3_commit_fn] 0-management: Add-brick commit failed.</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:20.693667] E [MSGID: 106123] [glusterd-mgmt-handler.c:616:glusterd_handle_commit_fn] 0-management: commit failed on operation Add brick</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Webserver11 log file:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="p1"><span class="s1">[2019-04-15 13:56:29.563270] I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req</span><o:p></o:p></p>
<p class="p1"><span class="s1">The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2019-04-15 13:56:29.563270] and [2019-04-15 13:56:29.566209]</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:33.996866] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x3a215) [0x7f36de924215] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe3e9d) [0x7f36de9cde9d] -->/lib64/libglusterfs.so.0(runner_log+0x115)
[0x7f36e9ed6ea5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=dockervols --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:33.996979] I [MSGID: 106578] [glusterd-brick-ops.c:1354:glusterd_op_perform_add_bricks] 0-management: replica-count is set 4</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:33.997004] I [MSGID: 106578] [glusterd-brick-ops.c:1364:glusterd_op_perform_add_bricks] 0-management: type is set 0, need to change it</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:34.013789] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: nfs already stopped</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:34.013849] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: nfs service is stopped</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:34.017535] I [MSGID: 106568] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 6087</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:35.018783] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: glustershd service is stopped</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:35.018952] I [MSGID: 106567] [glusterd-svc-mgmt.c:211:glusterd_svc_start] 0-management: Starting glustershd service</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:35.028306] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: bitd already stopped</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:35.028408] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: bitd service is stopped</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:35.028601] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: scrub already stopped</span><o:p></o:p></p>
<p class="p1"><span class="s1">[2019-04-15 14:00:35.028645] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: scrub service is stopped</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you for taking a look!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Boris<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Atin Mukherjee <atin.mukherjee83@gmail.com><br>
<b>Date: </b>Friday, April 12, 2019 at 1:10 PM<br>
<b>To: </b>Boris Goldowsky <bgoldowsky@cast.org><br>
<b>Cc: </b>Gluster-users <gluster-users@gluster.org><br>
<b>Subject: </b>Re: [Gluster-users] Volume stuck unable to add a brick<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Fri, 12 Apr 2019 at 22:32, Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org">bgoldowsky@cast.org</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I’ve got a replicated volume with three bricks (“1x3=3”), the idea is to have a common set of files that are locally available on all the machines (Scientific Linux 7, which is
essentially CentOS 7) in a cluster.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I tried to add on a fourth machine, so used a command like this:<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="m-2110851390084808688p1"><span class="m-2110851390084808688s1">sudo gluster volume add-brick dockervols replica 4 webserver8:/data/gluster/dockervols force</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">but the result is:<o:p></o:p></p>
<p class="m-2110851390084808688p1"><span class="m-2110851390084808688s1">volume add-brick: failed: Commit failed on webserver1. Please check log file for details.</span><o:p></o:p></p>
<p class="m-2110851390084808688p1"><span class="m-2110851390084808688s1">Commit failed on webserver8. Please check log file for details.</span><o:p></o:p></p>
<p class="m-2110851390084808688p1"><span class="m-2110851390084808688s1">Commit failed on webserver11. Please check log file for details.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: removing the new brick (this also fails) and trying again.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: checking the logs. The log files are not enlightening to me – I don’t know what’s normal and what’s not.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">From webserver8 & webserver11 could you attach glusterd log files?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Also please share following:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- gluster version? (gluster —version)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- Output of ‘gluster peer status’<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- Output of ‘gluster v info’ from all 4 nodes.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: deleting the brick directory from previous attempt, so that it’s not in the way.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: restarting gluster services<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: rebooting<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Tried: setting up a new volume, replicated to all four machines. This works, so I’m assuming it’s not a networking issue. But still fails with this existing volume that has the
critical data in it.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Running out of ideas. Any suggestions? Thank you!<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Boris<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</blockquote>
</div>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<p class="MsoNormal">--Atin<o:p></o:p></p>
</div>
</div>
</body>
</html>