<div dir="ltr">That makes sense. System is made of four data arrays with a hardware RAID 6 and then the distributed volume on top. I honestly don't know how that works, but the previous administrator said we had redundancy. I'm hoping there is a way to bypass the safeguard of migrating data when removing a brick from the volume, which in my beginner's mind, would be a straight-forward way of remedying the problem. Hopefully once the empty bricks are removed, the "missing" data will be visible again in the volume.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 27, 2019 at 3:59 PM Jim Kinney <<a href="mailto:jim.kinney@gmail.com">jim.kinney@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div style="text-align:left;direction:ltr"><div>Keep in mind that gluster is a metadata process. It doesn't really touch the actual volume files. The exception is the .glusterfs and .trashcan folders in the very top directory of the gluster volume.</div><div><br></div><div>When you create a gluster volume from brick, it doesn't format the filesystem. It uses what's already there.</div><div><br></div><div>So if you remove a volume and all it's bricks, you've not deleted data.</div><div><br></div><div>That said, if you are using anything but replicated bricks, which is what I use exclusively for my needs, then reassembling them into a new volume with correct name might be tricky. By listing the bricks in the exact same order as they were listed when creating the wrong name volume when making the correct named volume, it should use the same method to put data on the drives as previously and not scramble anything. </div><div><br></div><div>On Wed, 2019-02-27 at 14:24 -0500, Tami Greene wrote:</div><blockquote type="cite" style="margin:0px 0px 0px 0.8ex;border-left-width:2px;border-left-style:solid;border-left-color:rgb(114,159,207);padding-left:1ex"><div dir="ltr">I sent this and realized I hadn't registered. My apologies for the duplication<div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Subject: Added bricks with wrong name and now need to remove them without destroying volume.<br>To: <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br></div><br><br><div dir="ltr"><div dir="ltr"><br clear="all"><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">Yes, I broke it. Now I need help fixing it.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">I have an existing Gluster Volume, spread over 16 bricks and 4 servers; 1.5P space with 49% currently used . Added an additional 4 bricks and server as we expect large influx of data in the next 4 to 6 months. The system had been established by my predecessor, who is no longer here.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">First solo addition of bricks to gluster.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">Everything went smoothly until “gluster volume add-brick Volume newserver:/bricks/dataX/<a href="http://vol.name" target="_blank">vol.name</a>"<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> (I don’t have the exact response as I worked on this for almost 5 hours last night) Unable to add-brick as “it is already mounted” or something to that affect.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> Double checked my instructions, the name of the bricks. Everything seemed correct. Tried to add again adding “force.” Again, “unable to add-brick”<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> Because of the keyword (in my mind) “mounted” in the error, I checked /etc/fstab, where the name of the mount point is simply /bricks/dataX.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0);text-indent:0.5in"><span style="font-size:11pt">This convention was the same across all servers, so I thought I had discovered an error in my notes and changed the name to newserver:/bricks/dataX. <span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0);text-indent:0.5in"><span style="font-size:11pt">Still had to use force, but the bricks were added.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0);text-indent:0.5in"><span style="font-size:11pt">Restarted the gluster volume <a href="http://vol.name" target="_blank">vol.name</a>. No errors.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0);text-indent:0.5in"><span style="font-size:11pt">Rebooted; but /<a href="http://vol.name" target="_blank">vol.name</a> did not mount on reboot as the /etc/fstab instructs. So I attempted to mount manually and discovered a had a big mess on my hands.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> “Transport endpoint not connected” in addition to other messages.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> Discovered an issue between certificates and the auth.ssl-allow list because of the hostname of new server. I made correction and /<a href="http://vol.name" target="_blank">vol.name</a> mounted.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:11pt"> However, df -h indicated the 4 new bricks were not being seen as 400T were missing </span><span style="font-size:14.666666984558105px">from</span><span style="font-size:11pt"> what should have been available.</span><span style="font-size:11pt"></span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">Thankfully, I could add something to <a href="http://vol.name" target="_blank">vol.name</a> on one machine and see it on another machine and I wrongly assumed the volume was operational, even if the new bricks were not recognized.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">So I tried to correct the main issue by,<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> gluster volume remove <a href="http://vol.name" target="_blank">vol.name</a> newserver/bricks/dataX/<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> received prompt, data will be migrated before brick is removed continue (or something to that) and I started the process, think this won’t take long because there is no data.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> After 10 minutes and no apparent progress on the process, I did panic, thinking worse case scenario – it is writing zeros over my data.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> Executed the stop command and there was still no progress, and I assume it was due to no data on the brick to be remove causing the program to hang.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> Found the process ID and killed it.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><br></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><span style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:11pt">This morning, while all clients and servers can access /<a href="http://vol.name" target="_blank">vol.name</a>; not all of the data is present. I can find it under </span><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px">cluster, but users cannot reach it. I am, again, assume it is because of the 4 bricks that have been added, but aren't really a part of the volume because of their incorrect name.</span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">So – how do I proceed from here. <span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><br></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><span style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:11pt">1. Remove the 4 empty bricks from the volume </span><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px">without damaging data.</span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px">2. Correctly clear any metadata about these 4 bricks ONLY so they may be added correctly.</span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px"><br></span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px">If this doesn't restore the volume to full functionality, I'll write another post if I cannot find answer in the notes or on line.</span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><div><span style="font-size:11pt;color:rgb(0,0,0);font-family:Calibri,sans-serif">Tami</span>-- </div></div></div>
</div><br clear="all"><div><br></div><pre>_______________________________________________</pre><pre>Gluster-users mailing list</pre><a href="mailto:Gluster-users@gluster.org" target="_blank"><pre>Gluster-users@gluster.org</pre></a><pre><br></pre><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank"><pre>https://lists.gluster.org/mailman/listinfo/gluster-users</pre></a></div></div></blockquote><div><span><pre><pre>-- <br></pre>James P. Kinney III
Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain
<a href="http://heretothereideas.blogspot.com/" target="_blank">http://heretothereideas.blogspot.com/</a>
</pre></span></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Tami<br></div>