<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 11, 2018 at 7:38 PM, Jose Sanchez <span dir="ltr"><<a href="mailto:josesanc@carc.unm.edu" target="_blank">josesanc@carc.unm.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Hi Karthik</div><div><br></div><div>Looking at the information you have provided me, I would like to make sure that I’m running the right commands. </div><div><br></div><div>1. <span class="m_-734374313055314755Apple-tab-span" style="white-space:pre-wrap"> </span>gluster volume heal scratch info</div></div></blockquote><div>If the count is non zero, trigger the heal and wait for heal info count to become zero.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>2. <span class="m_-734374313055314755Apple-tab-span" style="white-space:pre-wrap"> </span>gluster volume remove-brick scratch <b>replica 1 </b><span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline">gluster02ib:/gdata/brick1/</span><span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline">sc<wbr>ratch <span style="font-variant-ligatures:no-common-ligatures;float:none;display:inline">gluster02ib:/gdata/<wbr>brick2/</span><span style="font-variant-ligatures:no-common-ligatures;float:none;display:inline">scratch </span></span>force</div></div></blockquote><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>3. <span class="m_-734374313055314755Apple-tab-span" style="white-space:pre-wrap"> </span>gluster volume add-brick<b> “#"</b> scratch <span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline">gluster02ib:/gdata/<wbr>brick1/</span><span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline">scratch <span style="font-variant-ligatures:no-common-ligatures;float:none;display:inline">gluster02ib:/<wbr>gdata/brick2/</span><span style="font-variant-ligatures:no-common-ligatures;float:none;display:inline">scratch</span></span></div><div><br></div><div><br></div><div>Based on the configuration I have, Brick 1 from Node A and B are tide together and Brick 2 from Node A and B are also tide together. Looking at your remove command (step #2), it seems that you want me to remove Brick 1 and 2 from Node B (gluster02ib). is that correct? I thought the data was distributed in bricks 1 between nodes A and B) and duplicated on Bricks 2 (node A and B). </div></div></blockquote><div>Data is duplicated between bricks 1 of nodes A & B and bricks 2 of nodes A & B and data is distributed between these two pairs.</div><div>You need not always remove the bricks 1 & 2 from node B itself. The idea here is to keep one copy from both the replica pairs.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>Also when I add the bricks back to gluster, do I need to specify if it is distributed or replicated?? and Do i need a configuration #?? for example on your command (Step #2) you have “replica 1” when remove bricks, do I need to do the same when adding the nodes back ?</div></div></blockquote><div>No. You just need to erase the data on those bricks and add those bricks back to the volume. The previous remove-brick command will make the volume plain distribute. Then simply adding the bricks without specifying any "#" will expand the volume as a plain distribute volue.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>Im planning on moving with this changes in few days. At this point each brick has 14tb and adding bricks 1 from node A and B, i have a total of 28tb, After doing all the process, (removing and adding bricks) I should be able to see a total of 56Tb right ?</div></div></blockquote><div>Yes after all these you will have 56TB in total.</div><div>After adding the bricks, do volume rebalance, so that the data which were present previously, will be moved to the correct bricks.</div><div><br></div><div>HTH,</div><div>Karthik</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>Thanks</div><div><br></div><div>Jose</div><div><br></div><div><br></div><div><br></div><br><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div>------------------------------<wbr>---</div><div>Jose Sanchez</div><span class=""><div>Systems/Network Analyst 1</div><div>Center of Advanced Research Computing</div><div><a href="https://maps.google.com/?q=1601+Central+Ave&entry=gmail&source=g">1601 Central Ave</a>.</div><div>MSC 01 1190</div><div>Albuquerque, NM 87131-0001</div><div><a href="http://carc.unm.edu" target="_blank">carc.unm.edu</a></div><div>575.636.4232</div></span></div>
</div><div><div class="h5">
<br><div><blockquote type="cite"><div>On Apr 7, 2018, at 8:29 AM, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>> wrote:</div><br class="m_-734374313055314755Apple-interchange-newline"><div><div dir="ltr">Hi Jose,<div><br></div><div>Thanks for providing the volume info. You have 2 subvolumes. Data is replicated within the bricks of that subvolumes.</div><div>First one consisting of Node A's brick1 & Node B's brick1 and the second one consisting of Node A's brick2 and Node B's brick2.</div><div>You don't have the same data on all the 4 bricks. Data are distributed between these two subvolumes.</div><div>To remove the replica you can use the command</div><div>gluster volume remove-brick scratch replica 1 <span style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline">gluster02ib:/gdata/brick1/</span><span style="font-style:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;text-decoration-style:initial;text-decoration-color:initial;color:rgb(77,47,45);font-family:Courier;font-size:14px;font-variant-ligatures:no-common-ligatures;background-color:rgb(223,219,195);float:none;display:inline">scra<wbr>tch <span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-style:normal;font-variant-ligatures:no-common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(223,219,195);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">gluster02ib:/gdata/brick2/</span><span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-style:normal;font-variant-ligatures:no-common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(223,219,195);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><wbr>scratch </span></span>force<span style="color:rgb(77,47,45);font-family:Courier;font-size:14px;font-style:normal;font-variant-ligatures:no-common-ligatures;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(223,219,195);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"></span></div><div>So you will have one copy of data present from both the distributes.</div><div>Before doing this make sure "gluster volume heal scratch info" value is zero. So copies you retain will have the correct data.</div><div>After the remove-brick erase the data from the backend.</div><div>Then you can expand the volume by following the steps at [1].</div><div><br></div><div>[1] <a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#expanding-volumes" target="_blank">https://docs.gluster.org/<wbr>en/latest/Administrator%<wbr>20Guide/Managing%20Volumes/#<wbr>expanding-volumes</a></div><div><br></div><div>Regards,</div><div>Karthik</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Apr 6, 2018 at 11:39 PM, Jose Sanchez <span dir="ltr"><<a href="mailto:josesanc@carc.unm.edu" target="_blank">josesanc@carc.unm.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Hi Karthik</div><div><br></div><div>this is our configuration, is 2x2 =4 , they are all replicated , each brick has 14tb. we have 2 nodes A and B, each one with brick 1 and 2. </div><div><br></div><div>Node A (replicated A1 (14tb) and B1 (14tb) ) same with node B (Replicated A2 (14tb) and B2 (14tb)).</div><div><br></div><div>Do you think we need to degrade the node first before removing it. i believe the same copy of data is on all 4 bricks, we would like to keep one of them, and add the other bricks as extra space</div><div><br></div><div>Thanks for your help on this</div><div><br></div><div>Jose</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br></div><div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">[root@gluster01 ~]# gluster volume info scratch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195);min-height:17px"><span style="font-variant-ligatures:no-common-ligatures"> </span><br class="m_-734374313055314755m_7358755837527423437webkit-block-placeholder"></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Volume Name: scratch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Type: Distributed-Replicate</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Volume ID: 23f1e4b1-b8e0-46c3-874a-58b472<wbr>8ea106</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Status: Started</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Snapshot Count: 0</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Number of Bricks: 2 x 2 = 4</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Transport-type: tcp,rdma</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Bricks:</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Brick1: gluster01ib:/gdata/brick1/scra<wbr>tch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Brick2: gluster02ib:/gdata/brick1/scra<wbr>tch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Brick3: gluster01ib:/gdata/brick2/scra<wbr>tch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Brick4: gluster02ib:/gdata/brick2/scra<wbr>tch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Options Reconfigured:</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">performance.readdir-ahead: on</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">nfs.disable: on</span></div></div><div><br></div><div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">[root@gluster01 ~]# gluster volume status all</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Status of volume: scratch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Gluster process TCP Port RDMA Port Online Pid</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>------------------------------<wbr>------------------</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Brick gluster01ib:/gdata/brick1/scra<wbr>tch 49152 49153 Y 1743 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Brick gluster02ib:/gdata/brick1/scra<wbr>tch 49156 49157 Y 1732 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Brick gluster01ib:/gdata/brick2/scra<wbr>tch 49154 49155 Y 1738 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Brick gluster02ib:/gdata/brick2/scra<wbr>tch 49158 49159 Y 1733 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Self-heal Daemon on localhost N/A N/A Y 1728 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Self-heal Daemon on gluster02ib N/A N/A Y 1726 </span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195);min-height:17px"><span style="font-variant-ligatures:no-common-ligatures"> </span><br class="m_-734374313055314755m_7358755837527423437webkit-block-placeholder"></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">Task Status of Volume scratch</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">------------------------------<wbr>------------------------------<wbr>------------------</span></div><div style="margin:0px;font-size:14px;line-height:normal;font-family:Courier;color:rgb(77,47,45);background-color:rgb(223,219,195)"><span style="font-variant-ligatures:no-common-ligatures">There are no active volume tasks</span></div></div><br><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div>------------------------------<wbr>---</div><div>Jose Sanchez</div><div>Systems/Network Analyst 1</div><span><div>Center of Advanced Research Computing</div><div><a href="https://maps.google.com/?q=1601+Central+Ave&entry=gmail&source=g" target="_blank">1601 Central Ave</a>.</div><div>MSC 01 1190</div><div>Albuquerque, NM 87131-0001</div><div><a href="http://carc.unm.edu/" target="_blank">carc.unm.edu</a></div><div>575.636.4232</div></span></div>
</div><div><div class="m_-734374313055314755h5">
<br><div><blockquote type="cite"><div>On Apr 6, 2018, at 3:49 AM, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>> wrote:</div><br class="m_-734374313055314755m_7358755837527423437Apple-interchange-newline"><div><div dir="ltr">Hi Jose,<div><br></div><div>By switching into pure distribute volume you will lose availability if something goes bad.</div><div><br></div><div>I am guessing you have a nX2 volume.</div><div>If you want to preserve one copy of the data in all the distributes, you can do that by decreasing the replica count in the remove-brick operation.</div><div>If you have any inconsistency, heal them first using the "gluster volume heal <volname>" command and wait till the</div><div>"gluster volume heal <volname> info" output becomes zero, before removing the bricks, so that you will have the correct data.</div><div>If you do not want to preserve the data then you can directly remove the bricks.</div><div>Even after removing the bricks the data will be present in the backend of the removed bricks. You have to manually erase them (both data and .glusterfs folder).</div><div><span style="color:rgb(34,34,34);font-family:arial,sans-serif;font-size:small;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">See [1] for more details on remove-brick.<span> </span></span><br></div><div><br></div><div>[1]. <a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Managing%20Volumes/#shrinking-volumes" target="_blank">https://docs.gluster.org/<wbr>en/latest/Administrator%20Guid<wbr>e/Managing%20Volumes/#shrinkin<wbr>g-volumes</a></div><div><br></div><div>HTH,</div><div>Karthik</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 5, 2018 at 8:17 PM, Jose Sanchez <span dir="ltr"><<a href="mailto:josesanc@carc.unm.edu" target="_blank">josesanc@carc.unm.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>We have a Gluster setup with 2 nodes (distributed replication) and we would like to switch it to the distributed mode. I know the data is duplicated between those nodes, what is the proper way of switching it to a distributed, we would like to double or gain the storage space on our gluster storage node. what happens with the data, do i need to erase one of the nodes? </div><div><br></div><div>Jose</div><div><br></div><div><br></div><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div>------------------------------<wbr>---</div><div>Jose Sanchez</div><div>Systems/Network Analyst </div><div>Center of Advanced Research Computing</div><div><a href="https://maps.google.com/?q=1601+Central+Ave&entry=gmail&source=g" target="_blank">1601 Central Ave</a>.</div><div>MSC 01 1190</div><div>Albuquerque, NM 87131-0001</div><div><a href="http://carc.unm.edu/" target="_blank">carc.unm.edu</a></div><div>575.636.4232</div></div>
</div>
<br></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div><br></div>
</div></blockquote></div><br></div></div></div></blockquote></div><br></div>
</div></blockquote></div><br></div></div></div></blockquote></div><br></div></div>