<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hello Karthik</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Im having trouble adding the two bricks back online. Any help is appreciated </div><div class=""><br class=""></div><div class="">thanks </div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">when i try to add-brick command this is what i get </div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@gluster01 ~]# gluster volume add-brick scratch gluster02ib:/gdata/brick2/scratch/</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">volume add-brick: failed: Pre Validation failed on gluster02ib. Brick: gluster02ib:/gdata/brick2/scratch not available. Brick may be containing or be contained by an existing brick</span></div></div><div class=""><br class=""></div><div class="">I have run the following commands and remove the .glusterfs hidden directories </div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@gluster02 ~]# setfattr -x trusted.glusterfs.volume-id /gdata/brick2/scratch/</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">setfattr: /gdata/brick2/scratch/: No such attribute</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@gluster02 ~]# setfattr -x trusted.gfid /gdata/brick2/scratch/</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">setfattr: /gdata/brick2/scratch/: No such attribute</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@gluster02 ~]# </span></div></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">this is what I get when I run status and info</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@gluster01 ~]# gluster volume info scratch</span></div><p style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195); min-height: 24px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> </span><br class="webkit-block-placeholder"></p><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Volume Name: scratch</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Type: Distribute</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Volume ID: 23f1e4b1-b8e0-46c3-874a-58b4728ea106</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Status: Started</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Snapshot Count: 0</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Number of Bricks: 4</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Transport-type: tcp,rdma</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Bricks:</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Brick1: gluster01ib:/gdata/brick1/scratch</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Brick2: gluster01ib:/gdata/brick2/scratch</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Brick3: gluster02ib:/gdata/brick1/scratch</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Brick4: gluster02ib:/gdata/brick2/scratch</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Options Reconfigured:</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">nfs.disable: on</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">performance.readdir-ahead: on</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@gluster01 ~]# </span></div></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@gluster02 ~]# gluster volume status scratch</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Status of volume: scratch</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Gluster process TCP Port RDMA Port Online Pid</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">------------------------------------------------------------------------------</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Brick gluster01ib:/gdata/brick1/scratch 49156 49157 Y 1819 </span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Brick gluster01ib:/gdata/brick2/scratch 49158 49159 Y 1827 </span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Brick gluster02ib:/gdata/brick1/scratch N/A N/A N N/A </span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Brick gluster02ib:/gdata/brick2/scratch N/A N/A N N/A </span></div><p style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195); min-height: 24px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> </span><br class="webkit-block-placeholder"></p><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Task Status of Volume scratch</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">------------------------------------------------------------------------------</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">There are no active volume tasks</span></div><p style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195); min-height: 24px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> </span><br class="webkit-block-placeholder"></p><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@gluster02 ~]# </span></div></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><br class=""></div><div class="">This are the logs files from Gluster ETC</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 20:56:54.390662] I [MSGID: 106143] [glusterd-pmap.c:250:pmap_registry_bind] 0-pmap: adding brick /gdata/brick1/scratch on port 49152</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 20:56:54.390798] I [MSGID: 106143] [glusterd-pmap.c:250:pmap_registry_bind] 0-pmap: adding brick /gdata/brick1/scratch.rdma on port 49153</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 20:56:54.391401] E [MSGID: 106005] [glusterd-utils.c:4877:glusterd_brick_start] 0-management: Unable to start brick gluster02ib:/gdata/brick1/scratch</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 20:56:54.391457] E [MSGID: 106074] [glusterd-brick-ops.c:2493:glusterd_op_add_brick] 0-glusterd: Unable to add bricks</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 20:56:54.391476] E [MSGID: 106123] [glusterd-mgmt.c:294:gd_mgmt_v3_commit_fn] 0-management: Add-brick commit failed.</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 20:56:54.391490] E [MSGID: 106123] [glusterd-mgmt-handler.c:603:glusterd_handle_commit_fn] 0-management: commit failed on operation Add brick</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 20:58:55.332262] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume scratch</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:02:07.464357] E [MSGID: 106452] [glusterd-utils.c:6064:glusterd_new_brick_validate] 0-management: Brick: gluster02ib:/gdata/brick1/scratch not available. Brick may be containing or be contained by an existing brick</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:02:07.464395] W [MSGID: 106122] [glusterd-mgmt.c:188:gd_mgmt_v3_pre_validate_fn] 0-management: ADD-brick prevalidation failed.</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:02:07.464414] E [MSGID: 106122] [glusterd-mgmt-handler.c:337:glusterd_handle_pre_validate_fn] 0-management: Pre Validation failed on operation Add brick</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:04:56.198662] E [MSGID: 106452] [glusterd-utils.c:6064:glusterd_new_brick_validate] 0-management: Brick: gluster02ib:/gdata/brick2/scratch not available. Brick may be containing or be contained by an existing brick</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:04:56.198700] W [MSGID: 106122] [glusterd-mgmt.c:188:gd_mgmt_v3_pre_validate_fn] 0-management: ADD-brick prevalidation failed.</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:04:56.198716] E [MSGID: 106122] [glusterd-mgmt-handler.c:337:glusterd_handle_pre_validate_fn] 0-management: Pre Validation failed on operation Add brick</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:07:11.084205] I [MSGID: 106482] [glusterd-brick-ops.c:447:__glusterd_handle_add_brick] 0-management: Received add brick req</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:07:11.087682] E [MSGID: 106452] [glusterd-utils.c:6064:glusterd_new_brick_validate] 0-management: Brick: gluster02ib:/gdata/brick2/scratch not available. Brick may be containing or be contained by an existing brick</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:07:11.087716] W [MSGID: 106122] [glusterd-mgmt.c:188:gd_mgmt_v3_pre_validate_fn] 0-management: ADD-brick prevalidation failed.</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:07:11.087729] E [MSGID: 106122] [glusterd-mgmt.c:884:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Add brick on local node</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:07:11.087741] E [MSGID: 106122] [glusterd-mgmt.c:2009:glusterd_mgmt_v3_initiate_all_phases] 0-management: Pre Validation Failed</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:12:22.340221] E [MSGID: 106452] [glusterd-utils.c:6064:glusterd_new_brick_validate] 0-management: Brick: gluster02ib:/gdata/brick2/scratch not available. Brick may be containing or be contained by an existing brick</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:12:22.340259] W [MSGID: 106122] [glusterd-mgmt.c:188:gd_mgmt_v3_pre_validate_fn] 0-management: ADD-brick prevalidation failed.</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:12:22.340274] E [MSGID: 106122] [glusterd-mgmt-handler.c:337:glusterd_handle_pre_validate_fn] 0-management: Pre Validation failed on operation Add brick</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[2018-04-25 21:18:13.427036] I [MSGID: 106499] [glusterd-handler.c:4349:__glusterd_handle_status_volume] 0-management: Received status volume req for volume scratch</span></div><div style="margin: 0px; font-size: 20px; line-height: normal; font-family: Courier; color: rgb(77, 47, 45); background-color: rgb(223, 219, 195);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">[root@gluster02 glusterfs]# </span></div></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><br class=""><div class="">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">---------------------------------</div><div class="">Jose Sanchez</div><div class="">Systems/Network Analyst 1</div><div class="">Center of Advanced Research Computing</div><div class="">1601 Central Ave.</div><div class="">MSC 01 1190</div><div class="">Albuquerque, NM 87131-0001</div><div class=""><a href="http://carc.unm.edu" class="">carc.unm.edu</a></div><div class="">575.636.4232</div></div>
</div>
<br class=""><div><blockquote type="cite" class=""><div class="">On Apr 12, 2018, at 12:11 AM, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" class="">ksubrahm@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Apr 11, 2018 at 7:38 PM, Jose Sanchez <span dir="ltr" class=""><<a href="mailto:josesanc@carc.unm.edu" target="_blank" class="">josesanc@carc.unm.edu</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">Hi Karthik</div><div class=""><br class=""></div><div class="">Looking at the information you have provided me, I would like to make sure that I’m running the right commands. </div><div class=""><br class=""></div><div class="">1. <span class="m_-734374313055314755Apple-tab-span" style="white-space:pre-wrap"> </span>gluster volume heal scratch info</div></div></blockquote><div class="">If the count is non zero, trigger the heal and wait for heal info count to become zero.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">2. <span class="m_-734374313055314755Apple-tab-span" style="white-space:pre-wrap"> </span>gluster volume remove-brick scratch <b class="">replica 1 </b><span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline" class="">gluster02ib:/gdata/brick1/</span><span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline" class="">sc<wbr class="">ratch <span style="font-variant-ligatures:no-common-ligatures;float:none;display:inline" class="">gluster02ib:/gdata/<wbr class="">brick2/</span><span style="font-variant-ligatures:no-common-ligatures;float:none;display:inline" class="">scratch </span></span>force</div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">3. <span class="m_-734374313055314755Apple-tab-span" style="white-space:pre-wrap"> </span>gluster volume add-brick<b class=""> “#"</b> scratch <span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline" class="">gluster02ib:/gdata/<wbr class="">brick1/</span><span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline" class="">scratch <span style="font-variant-ligatures:no-common-ligatures;float:none;display:inline" class="">gluster02ib:/<wbr class="">gdata/brick2/</span><span style="font-variant-ligatures:no-common-ligatures;float:none;display:inline" class="">scratch</span></span></div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">Based on the configuration I have, Brick 1 from Node A and B are tide together and Brick 2 from Node A and B are also tide together. Looking at your remove command (step #2), it seems that you want me to remove Brick 1 and 2 from Node B (gluster02ib). is that correct? I thought the data was distributed in bricks 1 between nodes A and B) and duplicated on Bricks 2 (node A and B). </div></div></blockquote><div class="">Data is duplicated between bricks 1 of nodes A & B and bricks 2 of nodes A & B and data is distributed between these two pairs.</div><div class="">You need not always remove the bricks 1 & 2 from node B itself. The idea here is to keep one copy from both the replica pairs.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><br class=""></div><div class="">Also when I add the bricks back to gluster, do I need to specify if it is distributed or replicated?? and Do i need a configuration #?? for example on your command (Step #2) you have “replica 1” when remove bricks, do I need to do the same when adding the nodes back ?</div></div></blockquote><div class="">No. You just need to erase the data on those bricks and add those bricks back to the volume. The previous remove-brick command will make the volume plain distribute. Then simply adding the bricks without specifying any "#" will expand the volume as a plain distribute volue.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><br class=""></div><div class="">Im planning on moving with this changes in few days. At this point each brick has 14tb and adding bricks 1 from node A and B, i have a total of 28tb, After doing all the process, (removing and adding bricks) I should be able to see a total of 56Tb right ?</div></div></blockquote><div class="">Yes after all these you will have 56TB in total.</div><div class="">After adding the bricks, do volume rebalance, so that the data which were present previously, will be moved to the correct bricks.</div><div class=""><br class=""></div><div class="">HTH,</div><div class="">Karthik</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><br class=""></div><div class="">Thanks</div><div class=""><br class=""></div><div class="">Jose</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div class="">
<div style="letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; word-wrap: break-word;" class=""><div class="">------------------------------<wbr class="">---</div><div class="">Jose Sanchez</div><span class=""><div class="">Systems/Network Analyst 1</div><div class="">Center of Advanced Research Computing</div><div class=""><a href="https://maps.google.com/?q=1601+Central+Ave&entry=gmail&source=g" class="">1601 Central Ave</a>.</div><div class="">MSC 01 1190</div><div class="">Albuquerque, NM 87131-0001</div><div class=""><a href="http://carc.unm.edu/" target="_blank" class="">carc.unm.edu</a></div><div class="">575.636.4232</div></span></div>
</div><div class=""><div class="h5">
<br class=""><div class=""><blockquote type="cite" class=""><div class="">On Apr 7, 2018, at 8:29 AM, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" target="_blank" class="">ksubrahm@redhat.com</a>> wrote:</div><br class="m_-734374313055314755Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Jose,<div class=""><br class=""></div><div class="">Thanks for providing the volume info. You have 2 subvolumes. Data is replicated within the bricks of that subvolumes.</div><div class="">First one consisting of Node A's brick1 & Node B's brick1 and the second one consisting of Node A's brick2 and Node B's brick2.</div><div class="">You don't have the same data on all the 4 bricks. Data are distributed between these two subvolumes.</div><div class="">To remove the replica you can use the command</div><div class="">gluster volume remove-brick scratch replica 1 <span style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline" class="">gluster02ib:/gdata/brick1/</span><span style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline" class="">scra<wbr class="">tch <span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-style:normal;font-variant-ligatures:no-common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(223,219,195);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline" class="">gluster02ib:/gdata/brick2/</span><span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-style:normal;font-variant-ligatures:no-common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(223,219,195);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline" class=""><wbr class="">scratch </span></span>force<span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-style:normal;font-variant-ligatures:no-common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(223,219,195);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline" class=""></span></div><div class="">So you will have one copy of data present from both the distributes.</div><div class="">Before doing this make sure "gluster volume heal scratch info" value is zero. So copies you retain will have the correct data.</div><div class="">After the remove-brick erase the data from the backend.</div><div class="">Then you can expand the volume by following the steps at [1].</div><div class=""><br class=""></div><div class="">[1] <a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#expanding-volumes" target="_blank" class="">https://docs.gluster.org/<wbr class="">en/latest/Administrator%<wbr class="">20Guide/Managing%20Volumes/#<wbr class="">expanding-volumes</a></div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Karthik</div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Fri, Apr 6, 2018 at 11:39 PM, Jose Sanchez <span dir="ltr" class=""><<a href="mailto:josesanc@carc.unm.edu" target="_blank" class="">josesanc@carc.unm.edu</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">Hi Karthik</div><div class=""><br class=""></div><div class="">this is our configuration, is 2x2 =4 , they are all replicated , each brick has 14tb. we have 2 nodes A and B, each one with brick 1 and 2. </div><div class=""><br class=""></div><div class="">Node A (replicated A1 (14tb) and B1 (14tb) ) same with node B (Replicated A2 (14tb) and B2 (14tb)).</div><div class=""><br class=""></div><div class="">Do you think we need to degrade the node first before removing it. i believe the same copy of data is on all 4 bricks, we would like to keep one of them, and add the other bricks as extra space</div><div class=""><br class=""></div><div class="">Thanks for your help on this</div><div class=""><br class=""></div><div class="">Jose</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[root@gluster01 ~]# gluster volume info scratch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195);min-height:17px" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><br class="m_-734374313055314755m_7358755837527423437webkit-block-placeholder"></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Volume Name: scratch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Type: Distributed-Replicate</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Volume ID: 23f1e4b1-b8e0-46c3-874a-58b472<wbr class="">8ea106</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Status: Started</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Snapshot Count: 0</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Number of Bricks: 2 x 2 = 4</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Transport-type: tcp,rdma</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Bricks:</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick1: gluster01ib:/gdata/brick1/scra<wbr class="">tch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick2: gluster02ib:/gdata/brick1/scra<wbr class="">tch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick3: gluster01ib:/gdata/brick2/scra<wbr class="">tch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick4: gluster02ib:/gdata/brick2/scra<wbr class="">tch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Options Reconfigured:</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">performance.readdir-ahead: on</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">nfs.disable: on</span></div></div><div class=""><br class=""></div><div class=""><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">[root@gluster01 ~]# gluster volume status all</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Status of volume: scratch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Gluster process TCP Port RDMA Port Online Pid</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">------------------------------<wbr class="">------------------------------<wbr class="">------------------</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick gluster01ib:/gdata/brick1/scra<wbr class="">tch 49152 49153 Y 1743 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick gluster02ib:/gdata/brick1/scra<wbr class="">tch 49156 49157 Y 1732 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick gluster01ib:/gdata/brick2/scra<wbr class="">tch 49154 49155 Y 1738 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Brick gluster02ib:/gdata/brick2/scra<wbr class="">tch 49158 49159 Y 1733 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Self-heal Daemon on localhost N/A N/A Y 1728 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Self-heal Daemon on gluster02ib N/A N/A Y 1726 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195);min-height:17px" class=""><span style="font-variant-ligatures:no-common-ligatures" class=""> </span><br class="m_-734374313055314755m_7358755837527423437webkit-block-placeholder"></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">Task Status of Volume scratch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">------------------------------<wbr class="">------------------------------<wbr class="">------------------</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)" class=""><span style="font-variant-ligatures:no-common-ligatures" class="">There are no active volume tasks</span></div></div><br class=""><div class="">
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" class=""><div class="">------------------------------<wbr class="">---</div><div class="">Jose Sanchez</div><div class="">Systems/Network Analyst 1</div><span class=""><div class="">Center of Advanced Research Computing</div><div class=""><a href="https://maps.google.com/?q=1601+Central+Ave&entry=gmail&source=g" target="_blank" class="">1601 Central Ave</a>.</div><div class="">MSC 01 1190</div><div class="">Albuquerque, NM 87131-0001</div><div class=""><a href="http://carc.unm.edu/" target="_blank" class="">carc.unm.edu</a></div><div class="">575.636.4232</div></span></div>
</div><div class=""><div class="m_-734374313055314755h5">
<br class=""><div class=""><blockquote type="cite" class=""><div class="">On Apr 6, 2018, at 3:49 AM, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" target="_blank" class="">ksubrahm@redhat.com</a>> wrote:</div><br class="m_-734374313055314755m_7358755837527423437Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Jose,<div class=""><br class=""></div><div class="">By switching into pure distribute volume you will lose availability if something goes bad.</div><div class=""><br class=""></div><div class="">I am guessing you have a nX2 volume.</div><div class="">If you want to preserve one copy of the data in all the distributes, you can do that by decreasing the replica count in the remove-brick operation.</div><div class="">If you have any inconsistency, heal them first using the "gluster volume heal <volname>" command and wait till the</div><div class="">"gluster volume heal <volname> info" output becomes zero, before removing the bricks, so that you will have the correct data.</div><div class="">If you do not want to preserve the data then you can directly remove the bricks.</div><div class="">Even after removing the bricks the data will be present in the backend of the removed bricks. You have to manually erase them (both data and .glusterfs folder).</div><div class=""><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline" class="">See [1] for more details on remove-brick.<span class=""> </span></span><br class=""></div><div class=""><br class=""></div><div class="">[1]. <a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#shrinking-volumes" target="_blank" class="">https://docs.gluster.org/<wbr class="">en/latest/Administrator%20Guid<wbr class="">e/Managing%20Volumes/#shrinkin<wbr class="">g-volumes</a></div><div class=""><br class=""></div><div class="">HTH,</div><div class="">Karthik</div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Thu, Apr 5, 2018 at 8:17 PM, Jose Sanchez <span dir="ltr" class=""><<a href="mailto:josesanc@carc.unm.edu" target="_blank" class="">josesanc@carc.unm.edu</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><br class=""></div><div class="">We have a Gluster setup with 2 nodes (distributed replication) and we would like to switch it to the distributed mode. I know the data is duplicated between those nodes, what is the proper way of switching it to a distributed, we would like to double or gain the storage space on our gluster storage node. what happens with the data, do i need to erase one of the nodes? </div><div class=""><br class=""></div><div class="">Jose</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word" class=""><div class="">------------------------------<wbr class="">---</div><div class="">Jose Sanchez</div><div class="">Systems/Network Analyst </div><div class="">Center of Advanced Research Computing</div><div class=""><a href="https://maps.google.com/?q=1601+Central+Ave&entry=gmail&source=g" target="_blank" class="">1601 Central Ave</a>.</div><div class="">MSC 01 1190</div><div class="">Albuquerque, NM 87131-0001</div><div class=""><a href="http://carc.unm.edu/" target="_blank" class="">carc.unm.edu</a></div><div class="">575.636.4232</div></div>
</div>
<br class=""></div><br class="">______________________________<wbr class="">_________________<br class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org" target="_blank" class="">Gluster-users@gluster.org</a><br class="">
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="">http://lists.gluster.org/mailm<wbr class="">an/listinfo/gluster-users</a><br class=""></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div>
</div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></div>
</div></blockquote></div><br class=""></body></html>