<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="auto">
<div>3.12.2
<div dir="auto"><br>
</div>
<div dir="auto">Boris</div>
<br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Apr 15, 2019 12:13 PM, Atin Mukherjee <atin.mukherjee83@gmail.com> wrote:<br type="attribution">
<blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div dir="ltr"><a href="mailto:ksubrahm@redhat.com">+Karthik Subrahmanya</a> <br>
<div><br>
</div>
<div>Didn't we we fix this problem recently? Failed to set extended attribute indicates that temp mount is failing and we don't have quorum number of bricks up.</div>
<div><br>
</div>
<div>Boris - What's the gluster version are you using?</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
<div class="elided-text">
<div dir="ltr">On Mon, Apr 15, 2019 at 7:35 PM Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org">bgoldowsky@cast.org</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb( 204 , 204 , 204 );padding-left:1ex">
<div>
<div>
<p>Atin, thank you for the reply. Here are all of those pieces of information:<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>[bgoldowsky@webserver9 ~]$ gluster --version<u></u><u></u></p>
<p>glusterfs 3.12.2<u></u><u></u></p>
<p>(same on all nodes)<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>[bgoldowsky@webserver9 ~]$ sudo gluster peer status<u></u><u></u></p>
<p>Number of Peers: 3<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Hostname: <a href="http://webserver11.cast.org">webserver11.cast.org</a><u></u><u></u></p>
<p>Uuid: c2b147fd-cab4-4859-9922-db5730f8549d<u></u><u></u></p>
<p>State: Peer in Cluster (Connected)<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Hostname: <a href="http://webserver1.cast.org">webserver1.cast.org</a><u></u><u></u></p>
<p>Uuid: 4b918f65-2c9d-478e-8648-81d1d6526d4c<u></u><u></u></p>
<p>State: Peer in Cluster (Connected)<u></u><u></u></p>
<p>Other names:<u></u><u></u></p>
<p>192.168.200.131<u></u><u></u></p>
<p>webserver1<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Hostname: <a href="http://webserver8.cast.org">webserver8.cast.org</a><u></u><u></u></p>
<p>Uuid: be2f568b-61c5-4016-9264-083e4e6453a2<u></u><u></u></p>
<p>State: Peer in Cluster (Connected)<u></u><u></u></p>
<p>Other names:<u></u><u></u></p>
<p>webserver8<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>[bgoldowsky@webserver1 ~]$ sudo gluster v info <u></u><u></u></p>
<p>Volume Name: dockervols<u></u><u></u></p>
<p>Type: Replicate<u></u><u></u></p>
<p>Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a<u></u><u></u></p>
<p>Status: Started<u></u><u></u></p>
<p>Snapshot Count: 0<u></u><u></u></p>
<p>Number of Bricks: 1 x 3 = 3<u></u><u></u></p>
<p>Transport-type: tcp<u></u><u></u></p>
<p>Bricks:<u></u><u></u></p>
<p>Brick1: webserver1:/data/gluster/dockervols<u></u><u></u></p>
<p>Brick2: webserver11:/data/gluster/dockervols<u></u><u></u></p>
<p>Brick3: webserver9:/data/gluster/dockervols<u></u><u></u></p>
<p>Options Reconfigured:<u></u><u></u></p>
<p>nfs.disable: on<u></u><u></u></p>
<p>transport.address-family: inet<u></u><u></u></p>
<p>auth.allow: 127.0.0.1<u></u><u></u></p>
<p> <u></u><u></u></p>
<p>Volume Name: testvol<u></u><u></u></p>
<p>Type: Replicate<u></u><u></u></p>
<p>Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332<u></u><u></u></p>
<p>Status: Started<u></u><u></u></p>
<p>Snapshot Count: 0<u></u><u></u></p>
<p>Number of Bricks: 1 x 4 = 4<u></u><u></u></p>
<p>Transport-type: tcp<u></u><u></u></p>
<p>Bricks:<u></u><u></u></p>
<p>Brick1: webserver1:/data/gluster/testvol<u></u><u></u></p>
<p>Brick2: webserver9:/data/gluster/testvol<u></u><u></u></p>
<p>Brick3: webserver11:/data/gluster/testvol<u></u><u></u></p>
<p>Brick4: webserver8:/data/gluster/testvol<u></u><u></u></p>
<p>Options Reconfigured:<u></u><u></u></p>
<p>transport.address-family: inet<u></u><u></u></p>
<p>nfs.disable: on<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>[bgoldowsky@webserver8 ~]$ sudo gluster v info<u></u><u></u></p>
<p>Volume Name: dockervols<u></u><u></u></p>
<p>Type: Replicate<u></u><u></u></p>
<p>Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a<u></u><u></u></p>
<p>Status: Started<u></u><u></u></p>
<p>Snapshot Count: 0<u></u><u></u></p>
<p>Number of Bricks: 1 x 3 = 3<u></u><u></u></p>
<p>Transport-type: tcp<u></u><u></u></p>
<p>Bricks:<u></u><u></u></p>
<p>Brick1: webserver1:/data/gluster/dockervols<u></u><u></u></p>
<p>Brick2: webserver11:/data/gluster/dockervols<u></u><u></u></p>
<p>Brick3: webserver9:/data/gluster/dockervols<u></u><u></u></p>
<p>Options Reconfigured:<u></u><u></u></p>
<p>nfs.disable: on<u></u><u></u></p>
<p>transport.address-family: inet<u></u><u></u></p>
<p>auth.allow: 127.0.0.1<u></u><u></u></p>
<p> <u></u><u></u></p>
<p>Volume Name: testvol<u></u><u></u></p>
<p>Type: Replicate<u></u><u></u></p>
<p>Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332<u></u><u></u></p>
<p>Status: Started<u></u><u></u></p>
<p>Snapshot Count: 0<u></u><u></u></p>
<p>Number of Bricks: 1 x 4 = 4<u></u><u></u></p>
<p>Transport-type: tcp<u></u><u></u></p>
<p>Bricks:<u></u><u></u></p>
<p>Brick1: webserver1:/data/gluster/testvol<u></u><u></u></p>
<p>Brick2: webserver9:/data/gluster/testvol<u></u><u></u></p>
<p>Brick3: webserver11:/data/gluster/testvol<u></u><u></u></p>
<p>Brick4: webserver8:/data/gluster/testvol<u></u><u></u></p>
<p>Options Reconfigured:<u></u><u></u></p>
<p>nfs.disable: on<u></u><u></u></p>
<p>transport.address-family: inet<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>[bgoldowsky@webserver9 ~]$ sudo gluster v info<u></u><u></u></p>
<p>Volume Name: dockervols<u></u><u></u></p>
<p>Type: Replicate<u></u><u></u></p>
<p>Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a<u></u><u></u></p>
<p>Status: Started<u></u><u></u></p>
<p>Snapshot Count: 0<u></u><u></u></p>
<p>Number of Bricks: 1 x 3 = 3<u></u><u></u></p>
<p>Transport-type: tcp<u></u><u></u></p>
<p>Bricks:<u></u><u></u></p>
<p>Brick1: webserver1:/data/gluster/dockervols<u></u><u></u></p>
<p>Brick2: webserver11:/data/gluster/dockervols<u></u><u></u></p>
<p>Brick3: webserver9:/data/gluster/dockervols<u></u><u></u></p>
<p>Options Reconfigured:<u></u><u></u></p>
<p>nfs.disable: on<u></u><u></u></p>
<p>transport.address-family: inet<u></u><u></u></p>
<p>auth.allow: 127.0.0.1<u></u><u></u></p>
<p> <u></u><u></u></p>
<p>Volume Name: testvol<u></u><u></u></p>
<p>Type: Replicate<u></u><u></u></p>
<p>Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332<u></u><u></u></p>
<p>Status: Started<u></u><u></u></p>
<p>Snapshot Count: 0<u></u><u></u></p>
<p>Number of Bricks: 1 x 4 = 4<u></u><u></u></p>
<p>Transport-type: tcp<u></u><u></u></p>
<p>Bricks:<u></u><u></u></p>
<p>Brick1: webserver1:/data/gluster/testvol<u></u><u></u></p>
<p>Brick2: webserver9:/data/gluster/testvol<u></u><u></u></p>
<p>Brick3: webserver11:/data/gluster/testvol<u></u><u></u></p>
<p>Brick4: webserver8:/data/gluster/testvol<u></u><u></u></p>
<p>Options Reconfigured:<u></u><u></u></p>
<p>nfs.disable: on<u></u><u></u></p>
<p>transport.address-family: inet<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>[bgoldowsky@webserver11 ~]$ sudo gluster v info<u></u><u></u></p>
<p>Volume Name: dockervols<u></u><u></u></p>
<p>Type: Replicate<u></u><u></u></p>
<p>Volume ID: 6093a9c6-ec6c-463a-ad25-8c3e3305b98a<u></u><u></u></p>
<p>Status: Started<u></u><u></u></p>
<p>Snapshot Count: 0<u></u><u></u></p>
<p>Number of Bricks: 1 x 3 = 3<u></u><u></u></p>
<p>Transport-type: tcp<u></u><u></u></p>
<p>Bricks:<u></u><u></u></p>
<p>Brick1: webserver1:/data/gluster/dockervols<u></u><u></u></p>
<p>Brick2: webserver11:/data/gluster/dockervols<u></u><u></u></p>
<p>Brick3: webserver9:/data/gluster/dockervols<u></u><u></u></p>
<p>Options Reconfigured:<u></u><u></u></p>
<p>auth.allow: 127.0.0.1<u></u><u></u></p>
<p>transport.address-family: inet<u></u><u></u></p>
<p>nfs.disable: on<u></u><u></u></p>
<p> <u></u><u></u></p>
<p>Volume Name: testvol<u></u><u></u></p>
<p>Type: Replicate<u></u><u></u></p>
<p>Volume ID: 4d5f00f5-00ea-4dcf-babf-1a76eca55332<u></u><u></u></p>
<p>Status: Started<u></u><u></u></p>
<p>Snapshot Count: 0<u></u><u></u></p>
<p>Number of Bricks: 1 x 4 = 4<u></u><u></u></p>
<p>Transport-type: tcp<u></u><u></u></p>
<p>Bricks:<u></u><u></u></p>
<p>Brick1: webserver1:/data/gluster/testvol<u></u><u></u></p>
<p>Brick2: webserver9:/data/gluster/testvol<u></u><u></u></p>
<p>Brick3: webserver11:/data/gluster/testvol<u></u><u></u></p>
<p>Brick4: webserver8:/data/gluster/testvol<u></u><u></u></p>
<p>Options Reconfigured:<u></u><u></u></p>
<p>transport.address-family: inet<u></u><u></u></p>
<p>nfs.disable: on<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>[bgoldowsky@webserver9 ~]$ sudo gluster volume add-brick dockervols replica 4 webserver8:/data/gluster/dockervols force<u></u><u></u></p>
<p>volume add-brick: failed: Commit failed on <a href="http://webserver8.cast.org">
webserver8.cast.org</a>. Please check log file for details.<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Webserver8 glusterd.log:<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>[2019-04-15 13:55:42.338197] I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<u></u><u></u></p>
<p>The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2019-04-15 13:55:42.338197] and [2019-04-15 13:55:42.341618]<u></u><u></u></p>
<p>[2019-04-15 14:00:20.445011] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x3a215) [0x7fe697764215] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe3e9d) [0x7fe69780de9d] -->/lib64/libglusterfs.so.0(runner_log+0x115)
[0x7fe6a2d16ea5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=dockervols --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd<u></u><u></u></p>
<p>[2019-04-15 14:00:20.445148] I [MSGID: 106578] [glusterd-brick-ops.c:1354:glusterd_op_perform_add_bricks] 0-management: replica-count is set 4<u></u><u></u></p>
<p>[2019-04-15 14:00:20.445184] I [MSGID: 106578] [glusterd-brick-ops.c:1364:glusterd_op_perform_add_bricks] 0-management: type is set 0, need to change it<u></u><u></u></p>
<p>[2019-04-15 14:00:20.672347] E [MSGID: 106054] [glusterd-utils.c:13863:glusterd_handle_replicate_brick_ops] 0-management: Failed to set extended attribute trusted.add-brick : Transport endpoint is not connected [Transport endpoint is not connected]<u></u><u></u></p>
<p>[2019-04-15 14:00:20.693491] E [MSGID: 101042] [compat.c:569:gf_umount_lazy] 0-management: Lazy unmount of /tmp/mntmvdFGq [Transport endpoint is not connected]<u></u><u></u></p>
<p>[2019-04-15 14:00:20.693597] E [MSGID: 106074] [glusterd-brick-ops.c:2590:glusterd_op_add_brick] 0-glusterd: Unable to add bricks<u></u><u></u></p>
<p>[2019-04-15 14:00:20.693637] E [MSGID: 106123] [glusterd-mgmt.c:312:gd_mgmt_v3_commit_fn] 0-management: Add-brick commit failed.<u></u><u></u></p>
<p>[2019-04-15 14:00:20.693667] E [MSGID: 106123] [glusterd-mgmt-handler.c:616:glusterd_handle_commit_fn] 0-management: commit failed on operation Add brick<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Webserver11 log file:<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>[2019-04-15 13:56:29.563270] I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req<u></u><u></u></p>
<p>The message "I [MSGID: 106488] [glusterd-handler.c:1559:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2019-04-15 13:56:29.563270] and [2019-04-15 13:56:29.566209]<u></u><u></u></p>
<p>[2019-04-15 14:00:33.996866] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0x3a215) [0x7f36de924215] -->/usr/lib64/glusterfs/3.12.2/xlator/mgmt/glusterd.so(+0xe3e9d) [0x7f36de9cde9d] -->/lib64/libglusterfs.so.0(runner_log+0x115)
[0x7f36e9ed6ea5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh --volname=dockervols --version=1 --volume-op=add-brick --gd-workdir=/var/lib/glusterd<u></u><u></u></p>
<p>[2019-04-15 14:00:33.996979] I [MSGID: 106578] [glusterd-brick-ops.c:1354:glusterd_op_perform_add_bricks] 0-management: replica-count is set 4<u></u><u></u></p>
<p>[2019-04-15 14:00:33.997004] I [MSGID: 106578] [glusterd-brick-ops.c:1364:glusterd_op_perform_add_bricks] 0-management: type is set 0, need to change it<u></u><u></u></p>
<p>[2019-04-15 14:00:34.013789] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: nfs already stopped<u></u><u></u></p>
<p>[2019-04-15 14:00:34.013849] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: nfs service is stopped<u></u><u></u></p>
<p>[2019-04-15 14:00:34.017535] I [MSGID: 106568] [glusterd-proc-mgmt.c:88:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 6087<u></u><u></u></p>
<p>[2019-04-15 14:00:35.018783] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: glustershd service is stopped<u></u><u></u></p>
<p>[2019-04-15 14:00:35.018952] I [MSGID: 106567] [glusterd-svc-mgmt.c:211:glusterd_svc_start] 0-management: Starting glustershd service<u></u><u></u></p>
<p>[2019-04-15 14:00:35.028306] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: bitd already stopped<u></u><u></u></p>
<p>[2019-04-15 14:00:35.028408] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: bitd service is stopped<u></u><u></u></p>
<p>[2019-04-15 14:00:35.028601] I [MSGID: 106132] [glusterd-proc-mgmt.c:84:glusterd_proc_stop] 0-management: scrub already stopped<u></u><u></u></p>
<p>[2019-04-15 14:00:35.028645] I [MSGID: 106568] [glusterd-svc-mgmt.c:243:glusterd_svc_stop] 0-management: scrub service is stopped<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Thank you for taking a look!<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>Boris<u></u><u></u></p>
<p><u></u> <u></u></p>
<p><u></u> <u></u></p>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb( 181 , 196 , 223 );padding:3pt 0in 0in">
<p><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">Atin Mukherjee <<a href="mailto:atin.mukherjee83@gmail.com">atin.mukherjee83@gmail.com</a>><br>
<b>Date: </b>Friday, April 12, 2019 at 1:10 PM<br>
<b>To: </b>Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org">bgoldowsky@cast.org</a>><br>
<b>Cc: </b>Gluster-users <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br>
<b>Subject: </b>Re: [Gluster-users] Volume stuck unable to add a brick<u></u><u></u></span></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
<div>
<div>
<p>On Fri, 12 Apr 2019 at 22:32, Boris Goldowsky <<a href="mailto:bgoldowsky@cast.org">bgoldowsky@cast.org</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb( 204 , 204 , 204 );padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p>I’ve got a replicated volume with three bricks (“1x3=3”), the idea is to have a common set of files that are locally available on all the machines (Scientific Linux 7, which is essentially CentOS 7) in a cluster.<u></u><u></u></p>
<p> <u></u><u></u></p>
<p>I tried to add on a fourth machine, so used a command like this:<u></u><u></u></p>
<p> <u></u><u></u></p>
<p>sudo gluster volume add-brick dockervols replica 4 webserver8:/data/gluster/dockervols force<u></u><u></u></p>
<p> <u></u><u></u></p>
<p>but the result is:<u></u><u></u></p>
<p>volume add-brick: failed: Commit failed on webserver1. Please check log file for details.<u></u><u></u></p>
<p>Commit failed on webserver8. Please check log file for details.<u></u><u></u></p>
<p>Commit failed on webserver11. Please check log file for details.<u></u><u></u></p>
<p> <u></u><u></u></p>
<p>Tried: removing the new brick (this also fails) and trying again.<u></u><u></u></p>
<p>Tried: checking the logs. The log files are not enlightening to me – I don’t know what’s normal and what’s not.<u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p>From webserver8 & webserver11 could you attach glusterd log files?<u></u><u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
<div>
<p>Also please share following:<u></u><u></u></p>
</div>
<div>
<p>- gluster version? (gluster —version)<u></u><u></u></p>
</div>
<div>
<p>- Output of ‘gluster peer status’<u></u><u></u></p>
</div>
<div>
<p>- Output of ‘gluster v info’ from all 4 nodes.<u></u><u></u></p>
</div>
<div>
<p><u></u> <u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb( 204 , 204 , 204 );padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p>Tried: deleting the brick directory from previous attempt, so that it’s not in the way.<u></u><u></u></p>
<p>Tried: restarting gluster services<u></u><u></u></p>
<p>Tried: rebooting<u></u><u></u></p>
<p>Tried: setting up a new volume, replicated to all four machines. This works, so I’m assuming it’s not a networking issue. But still fails with this existing volume that has the critical data in it.<u></u><u></u></p>
<p> <u></u><u></u></p>
<p>Running out of ideas. Any suggestions? Thank you!<u></u><u></u></p>
<p> <u></u><u></u></p>
<p>Boris<u></u><u></u></p>
<p> <u></u><u></u></p>
</div>
</div>
<p>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a><u></u><u></u></p>
</blockquote>
</div>
</div>
<p>-- <u></u><u></u></p>
<div>
<p>--Atin<u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</body>
</html>