<div dir="ltr">I sent this and realized I hadn't registered. My apologies for the duplication<div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Subject: Added bricks with wrong name and now need to remove them without destroying volume.<br>To: <<a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a>><br></div><br><br><div dir="ltr"><div dir="ltr"><br clear="all"><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">Yes, I broke it. Now I need help fixing it.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">I have an existing Gluster Volume, spread over 16 bricks and 4 servers; 1.5P space with 49% currently used . Added an additional 4 bricks and server as we expect large influx of data in the next 4 to 6 months. The system had been established by my predecessor, who is no longer here.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">First solo addition of bricks to gluster.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">Everything went smoothly until “gluster volume add-brick Volume newserver:/bricks/dataX/<a href="http://vol.name" target="_blank">vol.name</a>"<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> (I don’t have the exact response as I worked on this for almost 5 hours last night) Unable to add-brick as “it is already mounted” or something to that affect.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> Double checked my instructions, the name of the bricks. Everything seemed correct. Tried to add again adding “force.” Again, “unable to add-brick”<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> Because of the keyword (in my mind) “mounted” in the error, I checked /etc/fstab, where the name of the mount point is simply /bricks/dataX.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0);text-indent:0.5in"><span style="font-size:11pt">This convention was the same across all servers, so I thought I had discovered an error in my notes and changed the name to newserver:/bricks/dataX. <span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0);text-indent:0.5in"><span style="font-size:11pt">Still had to use force, but the bricks were added.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0);text-indent:0.5in"><span style="font-size:11pt">Restarted the gluster volume <a href="http://vol.name" target="_blank">vol.name</a>. No errors.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0);text-indent:0.5in"><span style="font-size:11pt">Rebooted; but /<a href="http://vol.name" target="_blank">vol.name</a> did not mount on reboot as the /etc/fstab instructs. So I attempted to mount manually and discovered a had a big mess on my hands.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> “Transport endpoint not connected” in addition to other messages.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> Discovered an issue between certificates and the auth.ssl-allow list because of the hostname of new server. I made correction and /<a href="http://vol.name" target="_blank">vol.name</a> mounted.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:11pt"> However, df -h indicated the 4 new bricks were not being seen as 400T were missing </span><span style="font-size:14.666666984558105px">from</span><span style="font-size:11pt"> what should have been available.</span><span style="font-size:11pt"></span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">Thankfully, I could add something to <a href="http://vol.name" target="_blank">vol.name</a> on one machine and see it on another machine and I wrongly assumed the volume was operational, even if the new bricks were not recognized.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">So I tried to correct the main issue by,<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> gluster volume remove <a href="http://vol.name" target="_blank">vol.name</a> newserver/bricks/dataX/<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> received prompt, data will be migrated before brick is removed continue (or something to that) and I started the process, think this won’t take long because there is no data.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> After 10 minutes and no apparent progress on the process, I did panic, thinking worse case scenario – it is writing zeros over my data.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> Executed the stop command and there was still no progress, and I assume it was due to no data on the brick to be remove causing the program to hang.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"> Found the process ID and killed it.<span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><br></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><span style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:11pt">This morning, while all clients and servers can access /<a href="http://vol.name" target="_blank">vol.name</a>; not all of the data is present. I can find it under </span><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px">cluster, but users cannot reach it. I am, again, assume it is because of the 4 bricks that have been added, but aren't really a part of the volume because of their incorrect name.</span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt">So – how do I proceed from here. <span></span></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><br></span></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><span style="color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:11pt">1. Remove the 4 empty bricks from the volume </span><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px">without damaging data.</span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px">2. Correctly clear any metadata about these 4 bricks ONLY so they may be added correctly.</span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px"><br></span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt"><font color="#000000" face="Calibri, sans-serif"><span style="font-size:14.666666984558105px">If this doesn't restore the volume to full functionality, I'll write another post if I cannot find answer in the notes or on line.</span></font></p><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:medium;font-family:Calibri,sans-serif;color:rgb(0,0,0)"><span style="font-size:11pt"><span> </span></span></p><div><span style="font-size:11pt;color:rgb(0,0,0);font-family:Calibri,sans-serif">Tami</span>-- </div></div></div>
</div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">Tami<br></div></div></div>