<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Mar 20, 2018 at 5:26 AM, Victor T <span dir="ltr"><<a href="mailto:hero_of_nothing_1@hotmail.com" target="_blank">hero_of_nothing_1@hotmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_7047255484402793044divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p style="margin-top:0;margin-bottom:0">That makes sense. In the case of "file damage," it would show up as files that could not be healed in logfiles or gluster volume heal [volume] info?</p></div></div></blockquote><div><br></div><div>If the damage affects more bricks than the volume redundancy, then probably yes. These files or directories will appear in "gluster volume heal <volname> info" permanently. In some cases, specially for directories, they could be manually healed. But this is always something that needs to be done with extra care and depends on each case, so I don't recommend to do it without help from someone that knows what is happening.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div id="m_7047255484402793044divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir="ltr"><p style="margin-top:0;margin-bottom:0"> Say we have acceptable backups, are there procedures to somehow overwrite
the bad gfid with a copy from a good backup?</p></div></div></blockquote><div><br></div><div>If a file is damaged but its parent directory is healthy, it should be possible to delete the file and then restore it from backup. All this should be done from the mount point, never directly on the bricks.</div><div><br></div><div>Xavi</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div id="m_7047255484402793044divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir="ltr">
</div>
<hr style="display:inline-block;width:98%">
<div id="m_7047255484402793044divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Xavi Hernandez <<a href="mailto:jahernan@redhat.com" target="_blank">jahernan@redhat.com</a>><br>
<b>Sent:</b> Monday, March 19, 2018 12:28:46 AM<div><div class="h5"><br>
<b>To:</b> Victor T<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Disperse volume recovery and healing</div></div></font>
<div> </div>
</div><div><div class="h5">
<div>
<div dir="ltr">Hi Victor,<br>
<div class="m_7047255484402793044x_gmail_extra"><br>
<div class="m_7047255484402793044x_gmail_quote">On Sun, Mar 18, 2018 at 3:47 AM, Victor T <span dir="ltr">
<<a href="mailto:hero_of_nothing_1@hotmail.com" target="_blank">hero_of_nothing_1@hotmail.com</a><wbr>></span> wrote:<br>
<blockquote class="m_7047255484402793044x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_7047255484402793044x_m_5053222459776498887divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif">
<span><font size="3" color="black" face="Calibri,Helvetica,sans-serif"><span id="m_7047255484402793044x_m_5053222459776498887m_8550796318300317070divtagdefaultwrapper" style="font-size:12pt">
<div id="m_7047255484402793044x_m_5053222459776498887m_8550796318300317070divtagdefaultwrapper"><font size="3" color="black" face="Calibri,Helvetica,sans-serif,EmojiFont,Apple Color Emoji,Segoe UI Emoji,NotoColorEmoji,Segoe UI Symbol,Android Emoji,EmojiSymbols"><span id="m_7047255484402793044x_m_5053222459776498887m_8550796318300317070divtagdefaultwrapper" style="font-size:12pt">
<div style="margin-top:0;margin-bottom:0"><br>
</div>
</span></font><font size="3" color="black" face="Calibri,Helvetica,sans-serif"><span id="m_7047255484402793044x_m_5053222459776498887m_8550796318300317070divtagdefaultwrapper" style="font-size:12pt"></span></font><i>No. After bringing up one brick and before stopping the next one,
you need to be sure that there are no damaged files. You shouldn't reboot a node if "gluster volume heal <volname> info" shows damaged files.</i><font size="3" color="black" face="Calibri,Helvetica,sans-serif"><span id="m_7047255484402793044x_m_5053222459776498887m_8550796318300317070divtagdefaultwrapper" style="font-size:12pt"></span></font></div>
</span></font></span>
<div><br>
What happens in this case then? I'm thinking about a situation where the servers are kept in an environment that we don't control - i.e. the cloud. If the VMs are forcibly rebooted without enough time to complete a heal before the next one goes down, then it
cannot be guaranteed that the data is safe? This has happened to me with Azure before, during the Meltdown/Spectre incident.<br>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>This is something that needs to be considered before deploying a dispersed volume (in fact any kind of volume). If multiple bricks can be restarted at *any* time, without *any* prior notification, then it's hard to guarantee much. You can think about this
as if it were similar to a RAID. If someone starts removing and adding disks without control, you will surely lose the entire volume if there is not enough time between disk removals to rebuild newly added or reconnected disks. In the case of Gluster you won't
lose the entire volume, but some files could get some damage.</div>
<div><br>
</div>
<div>If you can't control the sequence of reboots but you know when they will happen, the best thing you can do is to stop volume access (at least write access). This will prevent any corruption, even if multiple bricks are restarted at the same time. Note
that in this situation it's possible that you lose quorum, so the volume would be inaccessible anyway.</div>
<div><br>
</div>
<div>Xavi</div>
<div><br>
</div>
<blockquote class="m_7047255484402793044x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_7047255484402793044x_m_5053222459776498887divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif">
<div></div>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_7047255484402793044x_m_5053222459776498887divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Xavi Hernandez <<a href="mailto:jahernan@redhat.com" target="_blank">jahernan@redhat.com</a>><br>
<b>Sent:</b> Thursday, March 15, 2018 11:46:52 PM
<div>
<div class="m_7047255484402793044x_h5"><br>
<b>To:</b> Victor T<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Disperse volume recovery and healing</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="m_7047255484402793044x_h5">
<div>
<div dir="ltr">
<div class="m_7047255484402793044x_m_5053222459776498887x_gmail_extra">
<div class="m_7047255484402793044x_m_5053222459776498887x_gmail_quote">On Fri, Mar 16, 2018 at 4:57 AM, Victor T
<span dir="ltr"><<a href="mailto:hero_of_nothing_1@hotmail.com" target="_blank">hero_of_nothing_1@hotmail.com</a><wbr>></span> wrote:<br>
<blockquote class="m_7047255484402793044x_m_5053222459776498887x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif">
<div id="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0;margin-bottom:0">Xavi, does that mean that even if every node was rebooted one at a time even without issuing a heal that the volume would have no issues after running gluster volume heal [volname] when all bricks are back online?<br>
</p>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>No. After bringing up one brick and before stopping the next one, you need to be sure that there are no damaged files. You shouldn't reboot a node if "gluster volume heal <volname> info" shows damaged files.</div>
<div><br>
</div>
<div>The command "gluster volume heal <volname>" is only a tool to force heal to progress (until the bug is fixed).</div>
<div><br>
</div>
<div>Xavi</div>
<div><br>
</div>
<blockquote class="m_7047255484402793044x_m_5053222459776498887x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif">
<div id="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0;margin-bottom:0"></p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
<p style="margin-top:0;margin-bottom:0"><br>
</p>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070divRplyFwdMsg" dir="ltr"><font color="#000000" face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b> Xavi Hernandez <<a href="mailto:jahernan@redhat.com" target="_blank">jahernan@redhat.com</a>><br>
<b>Sent:</b> Thursday, March 15, 2018 12:09:05 AM<br>
<b>To:</b> Victor T<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Disperse volume recovery and healing</font>
<div> </div>
</div>
<div>
<div class="m_7047255484402793044x_m_5053222459776498887x_h5">
<div>
<div dir="ltr">Hi Victor,
<div><br>
</div>
<div class="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070x_gmail_extra">
<div class="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070x_gmail_quote">On Wed, Mar 14, 2018 at 12:30 AM, Victor T
<span dir="ltr"><<a href="mailto:hero_of_nothing_1@hotmail.com" id="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070LPlnk986324" target="_blank">hero_of_nothing_1@hotmail.com</a><wbr>></span> wrote:<br>
<blockquote class="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070x_gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070x_gmail-m_4006096338512438311divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<p style="margin-top:0px;margin-bottom:0px">I have a question about how disperse volumes handle brick failure. I'm running version 3.10.10 on all systems. If I have a disperse volume in a 4+2 configuration with 6 servers each serving 1 brick, and maintenance
needs to be performed on all systems, are there any general steps that need to be taken to ensure data is not lost or service interrupted? For example, can I just reboot each system sequentially after making sure sure the service is running on all servers
before rebooting the next system? Or is there a need to force/wait for a heal after each brick comes back online? If I have two bricks down for multiple days and then bring them back in, is there a need to issue a heal or something like a rebalance before
rebooting the other servers? There's lots of documentation about other volume types, but it seems information specific to dispersed volumes is a bit hard to find. Thanks a bunch.</p>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>On a 4+2 configuration you could bring down up to 2 bricks simultaneously for maintenance. However if something happens to one of the remaining 4 bricks, the volume would stop working. So in this case I would recommend to not have more than one server
down for maintenance at the same time unless the down time is very very small.</div>
<div><br>
</div>
<div>Once the stopped servers come back up again, you need to wait until all files are healed before proceeding with the next server. Failing to do so means that some files could have more than 2 non-healthy versions, what will make the file inaccessible until
enough healthy versions are available again.</div>
<div><br>
</div>
<div>Self-heal should be automatically triggered once the bricks come online, however there was a bug (<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1547662" id="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070LPlnk383013" target="_blank">https://bugzilla.redhat.com/s<wbr>how_bug.cgi?id=1547662</a>)
that could cause delays in the self-heal process. This bug should be fixed in the next version. Meantime you can force self-heal to progress by issuing "gluster volume heal <volname>" commands each time it seems to have stopped.</div>
<div><br>
</div>
<div>Once the output of "gluster volume heal <volname> info" reports 0 pending files on all bricks, you can proceed with the maintenance of the next server.</div>
<div><br>
</div>
<div>No need to do any rebalance for down bricks. Rebalance is basically needed when volume is expanded with more bricks.</div>
<div><br>
</div>
<div>Xavi</div>
<div><br>
</div>
<blockquote class="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070x_gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div id="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070x_gmail-m_4006096338512438311divtagdefaultwrapper" dir="ltr" style="font-size:12pt;color:rgb(0,0,0);font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" id="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070LPlnk828926" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" id="m_7047255484402793044x_m_5053222459776498887x_m_8550796318300317070LPlnk696927" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div></div></div>
</blockquote></div><br></div></div>