<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi,<div><br></div><div>Note: I guess the volume you are talking about is of type replica-2 (1x2). Usually replica 2 volumes are prone to split-brain. If you can consider converting them to arbiter or replica-3, they will handle most of the cases which can lead to slit-brains. For more information see [1].<br></div><div><br></div><div>Resolving the split-brain: [2] talks about how to interpret the heal info output and different ways to resolve them using the CLI/manually/using the favorite-child-policy.</div><div>If you are having entry split brain, and is a gfid split-brain (file/dir having different gfids on the replica bricks) then you can use the CLI option to resolve them. If a directory is in gfid split-brain in a distributed-replicate volume and you are using the source-brick option please make sure you use the brick of this subvolume, which has the same gfid as that of the other distribute subvolume(s) where you have the correct gfid, as the source.</div><div>If you are having a type mismatch then follow the steps in [3] to resolve the split-brain.</div><div><br></div><div>[1] <a href="https://docs.gluster.org/en/v3/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/">https://docs.gluster.org/en/v3/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/</a></div><div>[2] <a href="https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/">https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/</a></div><div>[3] <a href="https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/#dir-split-brain">https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/#dir-split-brain</a></div><div><br></div><div>HTH,</div><div>Karthik</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Mar 21, 2019 at 1:45 PM Milos Cuculovic <<a href="mailto:cuculovic@mdpi.com">cuculovic@mdpi.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;">I was now able to catch the split brain log:<div><br></div><div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">sudo gluster volume heal storage2 info</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Brick storage3:/data/data-cluster</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:256ca960-1601-4f0d-9b08-905c6fd52326> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">/dms/final_archive - Is in split-brain</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Status: Connected</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Number of entries: 3</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Brick storage4:/data/data-cluster</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0> </span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">/dms/final_archive - Is in split-brain</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Status: Connected</span></div><div style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures">Number of entries: 2</span></div><div><br></div><div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Milos</div></div><div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div><blockquote type="cite"><div>On 21 Mar 2019, at 09:07, Milos Cuculovic <<a href="mailto:cuculovic@mdpi.com" target="_blank">cuculovic@mdpi.com</a>> wrote:</div><br class="gmail-m_-1511781683842647198Apple-interchange-newline"><div><div>Since 24h, after upgrading from 4.0 to 4.1.7 one of the servers, the heal shows this:<br><br>sudo gluster volume heal storage2 info<br>Brick storage3:/data/data-cluster<br><gfid:256ca960-1601-4f0d-9b08-905c6fd52326> <br><gfid:7a63a729-c48f-4a00-9040-c3e2a0710ae6> <br>/dms/final_archive - Possibly undergoing heal<br><br>Status: Connected<br>Number of entries: 3<br><br>Brick storage4:/data/data-cluster<br><gfid:276fec9a-1c9b-4efe-9715-dcf4207e99b0> <br>/dms/final_archive - Possibly undergoing heal<br><br>Status: Connected<br>Number of entries: 2<br><br>The same files stay there. From time to time the status of the /dms/final_archive is in split brain at the following command shows:<br><br>sudo gluster volume heal storage2 info split-brain<br>Brick storage3:/data/data-cluster<br>/dms/final_archive<br>Status: Connected<br>Number of entries in split-brain: 1<br><br>Brick storage4:/data/data-cluster<br>/dms/final_archive<br>Status: Connected<br>Number of entries in split-brain: 1<br><br>How to know the file who is in split brain? The files in /dms/final_archive are not very important, fine to remove (ideally resolve the split brain) for the ones that differ.<br><br>I can only see the directory and GFID. Any idea on how to resolve this situation as I would like to continue with the upgrade on the 2nd server, and for this the heal needs to be done with 0 entries in sudo gluster volume heal storage2 info<br><br>Thank you in advance, Milos.</div></div></blockquote></div><br></div></div>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>