<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:"Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI Emoji";
panose-1:2 11 5 2 4 2 4 2 2 3;}
@font-face
{font-family:"\@Yu Gothic";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Vorformatiert Zchn";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLVorformatiertZchn
{mso-style-name:"HTML Vorformatiert Zchn";
mso-style-priority:99;
mso-style-link:"HTML Vorformatiert";
font-family:Consolas;}
span.E-MailFormatvorlage22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=DE link=blue vlink=purple style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-US>Hello Ilias,<br><br>in a fully healed state, each file in the glusterfs mount will be represented by one file in each brick of a replica set. Each of the files in the brick will be hardlinked to a gfid file that has the same name on all bricks.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>In a directory that is in split-brain or otherwise unhealable, either there is a file missing on one of the bricks, or there is a gfid mismatch where one copy of the file will link to a different gfid.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>If you delete just these files and their linked gfid files, nothing else is touched.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><div><p class=MsoNormal>Mit freundlichen Grüßen,<o:p></o:p></p><p class=MsoNormal>i.A. Thomas Bätzler<o:p></o:p></p><p class=MsoNormal>-- <o:p></o:p></p><p class=MsoNormal>BRINGE Informationstechnik GmbH<o:p></o:p></p><p class=MsoNormal>Zur Seeplatte 12<o:p></o:p></p><p class=MsoNormal>D-76228 Karlsruhe<o:p></o:p></p><p class=MsoNormal>Germany<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Fon: +49 721 94246-0<o:p></o:p></p><p class=MsoNormal>Fon: +49 171 5438457<o:p></o:p></p><p class=MsoNormal>Fax: +49 721 94246-66<o:p></o:p></p><p class=MsoNormal>Web: <a href="http://www.bringe.de/"><span style='color:#0563C1'>http://www.bringe.de/</span></a><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Geschäftsführer: Dipl.-Ing. (FH) Martin Bringe<o:p></o:p></p><p class=MsoNormal>Ust.Id: DE812936645, HRB 108943 Mannheim<o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b>Von:</b> Ilias Chasapakis forumZFD <chasapakis@forumZFD.de> <br><b>Gesendet:</b> Mittwoch, 17. August 2022 13:36<br><b>An:</b> Thomas Bätzler <t.baetzler@bringe.com>; gluster-users@gluster.org<br><b>Betreff:</b> Re: [Gluster-users] Directory in split brain does not heal - Gfs 9.2<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><p>Hi Thomas,<o:p></o:p></p><p>Thanks again for your replies and patience :)<o:p></o:p></p><p>We have also offline backups of the files.<o:p></o:p></p><p>So, just to verify I understood this correctly, deletion of a .glusterfs-gfid file doesn't inherently include the risk of the loss of the complete brick, right?<o:p></o:p></p><p>I saw you already applied this for your purposes so it worked for you... But just as a confirmation. Of course it is fully understood that the operational risk is on our side.<o:p></o:p></p><p>It is just an "information-wise" question :)<o:p></o:p></p><p>Best regards<br>Ilias<o:p></o:p></p><div><p class=MsoNormal>Am 17.08.22 um 12:47 schrieb Thomas Bätzler:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p class=MsoNormal><span lang=EN-US>Hello Ilias,<br><br>Please note that you can and should backup all of the file(s) involved in the split-brain by accessing them over the brick root instead of the gluster mount. That is also the reason why you’re not in danger of a failure cascade wiping out our data.</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US>Be careful when replacing bricks, though. You want that heal to go in the right direction </span><span lang=EN-US style='font-family:"Segoe UI Emoji",sans-serif'>😉</span><o:p></o:p></p><p class=MsoNormal><span lang=EN-US> </span><o:p></o:p></p><div><p class=MsoNormal>Mit freundlichen Grüßen,<o:p></o:p></p><p class=MsoNormal>i.A. Thomas Bätzler<o:p></o:p></p><p class=MsoNormal>-- <o:p></o:p></p><p class=MsoNormal>BRINGE Informationstechnik GmbH<o:p></o:p></p><p class=MsoNormal>Zur Seeplatte 12<o:p></o:p></p><p class=MsoNormal>D-76228 Karlsruhe<o:p></o:p></p><p class=MsoNormal>Germany<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Fon: +49 721 94246-0<o:p></o:p></p><p class=MsoNormal>Fon: +49 171 5438457<o:p></o:p></p><p class=MsoNormal>Fax: +49 721 94246-66<o:p></o:p></p><p class=MsoNormal>Web: <a href="http://www.bringe.de/"><span style='color:#0563C1'>http://www.bringe.de/</span></a><o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Geschäftsführer: Dipl.-Ing. (FH) Martin Bringe<o:p></o:p></p><p class=MsoNormal>Ust.Id: DE812936645, HRB 108943 Mannheim<o:p></o:p></p></div><p class=MsoNormal> <o:p></o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b>Von:</b> Gluster-users <a href="mailto:gluster-users-bounces@gluster.org"><gluster-users-bounces@gluster.org></a> <b>Im Auftrag von </b>Ilias Chasapakis forumZFD<br><b>Gesendet:</b> Mittwoch, 17. August 2022 11:18<br><b>An:</b> <a href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a><br><b>Betreff:</b> Re: [Gluster-users] Directory in split brain does not heal - Gfs 9.2<o:p></o:p></p></div></div><p class=MsoNormal> <o:p></o:p></p><p>Thanks for the suggestions. My question is if the risk is actually related to only losing the file/dir or actually creating inconsistencies that span through the bricks and "break everything".<br>Of course we have to take action anyway for this not to spread (as we already now have a second entry that developed an "unhealable" directory split-brain) so it is just a question of evaluation before acting.<o:p></o:p></p><div><p class=MsoNormal>Am 12.08.22 um 18:12 schrieb Thomas Bätzler:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>Am 12.08.2022 um 17:12 schrieb Ilias Chasapakis forumZFD:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p>Dear fellow gluster users,<o:p></o:p></p><p>we are facing a problem with our replica 3 setup. Glusterfs version is 9.2.<o:p></o:p></p><p>We have a problem with a directory that is in split-brain and we cannot manage to heal with:<o:p></o:p></p><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><p>gluster volume heal gfsVol split-brain latest-mtime /folder<o:p></o:p></p></blockquote><p>The command throws the following error: "failed:Transport endpoint is not connected." <o:p></o:p></p><p>So the split brain directory entry remains and and so the whole healing process is not completing and other entries get stuck.<o:p></o:p></p><p>I saw there is a python script available <a href="https://github.com/joejulian/glusterfs-splitbrain" target="_blank">https://github.com/joejulian/glusterfs-splitbrain</a> Would that be a good solution to try? To be honest we are a bit concerned with deleting the gfid and the files from the brick manually as it seems it can create inconsistencies and break things... I can of course give you more information about our setup and situation, but if you already have some tip, that would be fantastic.<o:p></o:p></p></blockquote><p>You could at least verify what's going on: Go to your brick roots and list /folder from each. You have 3n bricks with n replica sets. Find the replica set where you can spot a difference. It's most likely a file or directory that's missing or different. If it's a file, do a ls -ain on the file on each brick in the replica set. It'll report an inode number. Do a find .glusterfs -inum from the brick root. You'll likely see that you have different gfid-files.<o:p></o:p></p><p>To fix the problem, you have to help gluster along by cleaning up the mess. This is completely "do it at your own risk, it worked for me, ymmv": cp (not mv!) a copy of the file you want to keep. On each brick in the replica-set, delete the gfid-file and the datafile. Try a heal on the volume and verify that you can access the path in question using the glusterfs mount. Copy back your salvaged file using the glusterfs mount.<o:p></o:p></p><p>We had this happening quite often on a heavily loaded glusterfs shared filesystem that held a mail-spool. There would be parallel accesses trying to mv files and sometimes we'd end up with mismatched data on the bricks of the replica set. I've reported this on github, but apparently it wasn't seen as a serious problem. We've moved on to ceph FS now. That sure has bugs, too, but hopefully not as aggravating.<o:p></o:p></p><pre>MfG,<o:p></o:p></pre><pre>i.A. Thomas Bätzler<o:p></o:p></pre><pre>-- <o:p></o:p></pre><pre>BRINGE Informationstechnik GmbH<o:p></o:p></pre><pre>Zur Seeplatte 12<o:p></o:p></pre><pre>D-76228 Karlsruhe<o:p></o:p></pre><pre>Germany<o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Fon: +49 721 94246-0<o:p></o:p></pre><pre>Fon: +49 171 5438457<o:p></o:p></pre><pre>Fax: +49 721 94246-66<o:p></o:p></pre><pre>Web: <a href="http://www.bringe.de/">http://www.bringe.de/</a><o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Geschäftsführer: Dipl.-Ing. (FH) Martin Bringe<o:p></o:p></pre><pre>Ust.Id: DE812936645, HRB 108943 Mannheim<o:p></o:p></pre><p class=MsoNormal><br><br><br><o:p></o:p></p><pre>________<o:p></o:p></pre><pre> <o:p></o:p></pre><pre> <o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Community Meeting Calendar:<o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Schedule -<o:p></o:p></pre><pre>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<o:p></o:p></pre><pre>Bridge: <a href="https://meet.google.com/cpu-eiue-hvk">https://meet.google.com/cpu-eiue-hvk</a><o:p></o:p></pre><pre>Gluster-users mailing list<o:p></o:p></pre><pre><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><o:p></o:p></pre><pre><a href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></pre></blockquote><pre>-- <o:p></o:p></pre><pre><span style='font-family:"Tahoma",sans-serif'></span>forumZFD<o:p></o:p></pre><pre>Entschieden für Frieden | Committed to Peace<o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Ilias Chasapakis<o:p></o:p></pre><pre>Referent IT | IT Consultant<o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service<o:p></o:p></pre><pre>Am Kölner Brett 8 | 50825 Köln | Germany<o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Tel 0221 91273243 | Fax 0221 91273299 | <a href="http://www.forumZFD.de">http://www.forumZFD.de</a><o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Vorstand nach § 26 BGB, einzelvertretungsberechtigt | Executive Board:<o:p></o:p></pre><pre>Oliver Knabe (Vorsitz | Chair), Jens von Bargen, Alexander Mauz<o:p></o:p></pre><pre>VR 17651 Amtsgericht Köln<o:p></o:p></pre><pre> <o:p></o:p></pre><pre>Spenden | Donations: IBAN DE37 3702 0500 0008 2401 01 BIC BFSWDE33XXX<o:p></o:p></pre></blockquote><pre>-- <o:p></o:p></pre><pre><span style='font-family:"Tahoma",sans-serif'></span>forumZFD<o:p></o:p></pre><pre>Entschieden für Frieden | Committed to Peace<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Ilias Chasapakis<o:p></o:p></pre><pre>Referent IT | IT Consultant<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Forum Ziviler Friedensdienst e.V. | Forum Civil Peace Service<o:p></o:p></pre><pre>Am Kölner Brett 8 | 50825 Köln | Germany<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Tel 0221 91273243 | Fax 0221 91273299 | <a href="http://www.forumZFD.de">http://www.forumZFD.de</a><o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Vorstand nach § 26 BGB, einzelvertretungsberechtigt | Executive Board:<o:p></o:p></pre><pre>Oliver Knabe (Vorsitz | Chair), Jens von Bargen, Alexander Mauz<o:p></o:p></pre><pre>VR 17651 Amtsgericht Köln<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Spenden | Donations: IBAN DE37 3702 0500 0008 2401 01 BIC BFSWDE33XXX<o:p></o:p></pre></div></body></html>