<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div class="">Just for other users.. they may find this usefull.</div><div class=""><br class=""></div>I finally started Gluster server process on failed node that lost brick and all went OK.<div class="">Server is again available as a peer and failed brick is not running, so I can continue with replace brick/ reset brick operation. </div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On 16 Apr 2019, at 17:44, Martin Toth <<a href="mailto:snowmailer@gmail.com" class="">snowmailer@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="Content-Type" content="text/html; charset=utf-8" class=""><div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Thanks for clarification, one more question.<div class=""><br class=""></div><div class="">When I will recover(boot) failed node back and this peer will be available again to remaining two nodes. How do I tell gluster to mark this brick as failed ?</div><div class=""><br class=""></div><div class="">I mean, I’ve booted failed node back without networking. Disk partition (ZFS pool on another disks) where brick was before failure is lost.</div><div class="">Now I can start gluster event when I don't have ZFS pool where failed brick was before ?</div><div class=""><br class=""></div><div class="">This wont be a problem when I will connect this node back to cluster ? (before brick replace/reset command will be issued)</div><div class=""><br class=""></div><div class="">Thanks. BR!</div><div class="">Martin</div><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 11 Apr 2019, at 15:40, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" class="">ksubrahm@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><br class="Apple-interchange-newline"><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div class="gmail_quote" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div dir="ltr" class="gmail_attr">On Thu, Apr 11, 2019 at 6:38 PM Martin Toth <<a href="mailto:snowmailer@gmail.com" class="">snowmailer@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="overflow-wrap: break-word;" class="">Hi Karthik,<div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 11, 2019 at 12:43 PM Martin Toth <<a href="mailto:snowmailer@gmail.com" target="_blank" class="">snowmailer@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="">Hi Karthik,<div class=""><br class=""></div><div class="">more over, I would like to ask if there are some recommended settings/parameters for SHD in order to achieve good or fair I/O while volume will be healed when I will replace Brick (this should trigger healing process). </div></div></blockquote><div class="">If I understand you concern correctly, you need to get fair I/O performance for clients while healing takes place as part of the replace brick operation. For this you can turn off the "data-self-heal" and "metadata-self-heal" options until the heal completes on the new brick.</div></div></div></div></blockquote><div class=""><br class=""></div><div class="">This is exactly what I mean. I am running VM disks on remaining 2 (out of 3 - one failed as mentioned) nodes and I need to ensure there will be fair I/O performance available on these two nodes while replace brick operation will heal volume.</div><div class="">I will not run any VMs on node where replace brick operation will be running. So if I understand correctly, when I will set :</div><div class=""><br class=""></div><div class=""># gluster volume set <volname> cluster.data-self-heal off<div class=""># gluster volume set <volname> cluster.metadata-self-heal off</div></div><div class=""><br class=""></div><div class="">this will tell Gluster clients (libgfapi and FUSE mount) not to read from node “where replace brick operation” is in place but from remaing two healthy nodes. Is this correct ? Thanks for clarification.</div></div></div></div></blockquote><div class="">The reads will be served from one of the good bricks since the file will either be not present on the replaced brick at the time of read or it will be present but marked for heal if it is not already healed. If already healed by SHD, then it could be served from the new brick as well, but there won't be any problem in reading from there in that scenario.</div><div class="">By setting these two options whenever a read comes from client it will not try to heal the file for data/metadata. Otherwise it would try to heal (if not already healed by SHD) when the read comes on this, hence slowing down the client.</div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="overflow-wrap: break-word;" class=""><div class=""><div class=""><div class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote">Turning off client side healing doesn't compromise data integrity and consistency. During the read request from client, pending xattr is evaluated for replica copies and read is only served from correct copy. During writes, IO will continue on both the replicas, SHD will take care of healing files.</div><div class="gmail_quote">After replacing the brick, we strongly recommend you to consider upgrading your gluster to one of the maintained versions. We have many stability related fixes there, which can handle some critical issues and corner cases which you could hit during these kind of scenarios.</div></div></div></blockquote><div class=""><br class=""></div><div class="">This will be first priority in infrastructure after fixing this cluster back to fully functional replica3. I will upgrade to 3.12.x and then to version 5 or 6.</div></div></div></div></blockquote><div class="">Sounds good.</div><div class=""><br class=""></div><div class="">If you are planning to have the same name for the new brick and if you get the error like "Brick may be containing or be contained by an existing brick" even after using the force option, try using a different name. That should work.</div><div class=""><br class=""></div><div class="">Regards,</div><div class="">Karthik </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div style="overflow-wrap: break-word;" class=""><div class=""><div class=""><div class=""><br class=""></div><div class="">BR, </div><div class="">Martin</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="gmail_quote">Regards,</div><div class="gmail_quote">Karthik<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class=""><div class="">I had some problems in past when healing was triggered, VM disks became unresponsive because healing took most of I/O. My volume containing only big files with VM disks.</div><div class=""><br class=""></div><div class="">Thanks for suggestions.</div><div class="">BR, </div><div class="">Martin</div><div class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 10 Apr 2019, at 12:38, Martin Toth <<a href="mailto:snowmailer@gmail.com" target="_blank" class="">snowmailer@gmail.com</a>> wrote:</div><br class="gmail-m_1864269736194344118m_-3413167073120700099gmail-m_-6004318533745557497Apple-interchange-newline"><div class=""><div class="">Thanks, this looks ok to me, I will reset brick because I don't have any data anymore on failed node so I can use same path / brick name.<div class=""><br class=""></div><div class="">Is reseting brick dangerous command? Should I be worried about some possible failure that will impact remaining two nodes? I am running really old 3.7.6 but stable version.</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">BR!</div><div class=""><br class=""></div><div class="">Martin<br class=""><div class=""> <br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On 10 Apr 2019, at 12:20, Karthik Subrahmanya <<a href="mailto:ksubrahm@redhat.com" target="_blank" class="">ksubrahm@redhat.com</a>> wrote:</div><br class="gmail-m_1864269736194344118m_-3413167073120700099gmail-m_-6004318533745557497Apple-interchange-newline"><div class=""><div dir="ltr" class="">Hi Martin,<div class=""><br class=""></div><div class="">After you add the new disks and creating raid array, you can run the following command to replace the old brick with new one:</div><div class=""><br class=""></div><div class="">- If you are going to use a different name to the new brick you can run</div><div class="">gluster volume replace-brick <volname> <old-brick> <new-brick> commit force</div><div class=""><br class=""></div><div class="">- If you are planning to use the same name for the new brick as well then you can use</div><div class="">gluster volume reset-brick <volname> <old-brick> <new-brick> commit force</div><div class="">Here old-brick & new-brick's hostname & path should be same.</div><div class=""><br class=""></div><div class="">After replacing the brick, make sure the brick comes online using volume status.</div><div class="">Heal should automatically start, you can check the heal status to see all the files gets replicated to the newly added brick. If it does not start automatically, you can manually start that by running gluster volume heal <volname>.</div><div class=""><br class=""></div><div class="">HTH,</div><div class="">Karthik</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 10, 2019 at 3:13 PM Martin Toth <<a href="mailto:snowmailer@gmail.com" target="_blank" class="">snowmailer@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">Hi all,<br class=""><br class="">I am running replica 3 gluster with 3 bricks. One of my servers failed - all disks are showing errors and raid is in fault state.<br class=""><br class="">Type: Replicate<br class="">Volume ID: 41d5c283-3a74-4af8-a55d-924447bfa59a<br class="">Status: Started<br class="">Number of Bricks: 1 x 3 = 3<br class="">Transport-type: tcp<br class="">Bricks:<br class="">Brick1: node1.san:/tank/gluster/gv0imagestore/brick1<br class="">Brick2: node2.san:/tank/gluster/gv0imagestore/brick1 <— this brick is down<br class="">Brick3: node3.san:/tank/gluster/gv0imagestore/brick1<br class=""><br class="">So one of my bricks is totally failed (node2). It went down and all data are lost (failed raid on node2). Now I am running only two bricks on 2 servers out from 3.<br class="">This is really critical problem for us, we can lost all data. I want to add new disks to node2, create new raid array on them and try to replace failed brick on this node.<span class="Apple-converted-space"> </span><br class=""><br class="">What is the procedure of replacing Brick2 on node2, can someone advice? I can’t find anything relevant in documentation.<br class=""><br class="">Thanks in advance,<br class="">Martin<br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" target="_blank" class="">Gluster-users@gluster.org</a><br class=""><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></div></blockquote></div></div></div></div></div></blockquote></div></div></div></blockquote></div></div></div></blockquote></div></div></div></blockquote></div></div></blockquote></div><br class=""></div></div></div></blockquote></div><br class=""></div></body></html>