<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 23 January 2017 at 20:04, Gambit15 <span dir="ltr"><<a href="mailto:dougti+gluster@gmail.com" target="_blank">dougti+gluster@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Have you verified that Gluster has marked the files as split-brain?<br></div></div></div></div></blockquote><div><br></div><div>Gluster does not recognise all the files as split-brain - in fact only a handful are recognised as such - e.g for the example I pasted, its not listed - however the gfid is different (I believe this should be the same?)</div><div><br></div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><br></div>gluster volume heal <vol> info split-brain<br><br></div>If you're fairly confident about which files are correct, you can automate the split-brain healing procedure.<br><br></div>From the manual...<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> volume heal <VOLNAME> split-brain bigger-file <FILE><br> Performs healing of <FILE> which is in split-brain by choosing the bigger file in the replica as source.<br><br> volume heal <VOLNAME> split-brain source-brick <HOSTNAME:BRICKNAME><br> Selects <HOSTNAME:BRICKNAME> as the source for all the files that are in split-brain in that replica and heals them.<br><br> volume heal <VOLNAME> split-brain source-brick <HOSTNAME:BRICKNAME> <FILE><br> Selects the split-brained <FILE> present in <HOSTNAME:BRICKNAME> as source and completes heal.<br></blockquote><div><br></div><div>D <br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On 23 January 2017 at 16:28, James Wilkins <span dir="ltr"><<a href="mailto:nebulai@gmail.com" target="_blank">nebulai@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr">Hello,<div><br></div><div>I have a couple of gluster clusters - setup with distributed/replicated volumes that have starting incrementing the heal-count from statistics - and for some files returning input/output error when attempting to access said files from a fuse mount.</div><div><br></div><div>If i take one volume, from one cluster as an example:</div><div><br></div><div>gluster volume heal storage01 statistics info<br></div><div><snip></div><div><div>Brick storage02.<redacted>:/storage/<wbr>sdc/brick_storage01</div><div>Number of entries: 595</div></div><div></snip></div><div><br></div><div>And then proceed to look at one of these files (have found 2 copies - one on each server / brick)</div><div><br></div><div>First brick:</div><div><br></div><div><div># getfattr -m . -d -e hex /storage/sdc/brick_storage01/<wbr>projects/183-57c559ea4d60e-can<wbr>ary-test--node02/wordpress285-<wbr>data/html/wp-content/themes/<wbr>twentyfourteen/single.php</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: storage/sdc/brick_storage01/pr<wbr>ojects/183-57c559ea4d60e-canar<wbr>y-test--node02/wordpress285-<wbr>data/html/wp-content/themes/<wbr>twentyfourteen/single.php</div><div>security.selinux=0x73797374656<wbr>d5f753a6f626a6563745f723a676c7<wbr>573746572645f627269636b5f743a7<wbr>33000</div><div>trusted.afr.dirty=0x0000000000<wbr>00000000000000</div><div>trusted.afr.storage01-client-0<wbr>=0x000000020000000100000000</div><div>trusted.bit-rot.version=0x0200<wbr>0000000000005874e2cd0000459d</div><div>trusted.gfid=0xda4253be1c2647b<wbr>7b6ec5c045d61d216</div><div>trusted.glusterfs.quota.c97648<wbr>26-596a-4886-9bc0-60ee9b3fce44<wbr>.contri.1=0x000000000000060000<wbr>00000000000001</div><div>trusted.pgfid.c9764826-596a-48<wbr>86-9bc0-60ee9b3fce44=0x0000000<wbr>1</div></div><div><br></div><div>Second Brick:</div><div><br></div><div><div># getfattr -m . -d -e hex /storage/sdc/brick_storage01/p<wbr>rojects/183-57c559ea4d60e-cana<wbr>ry-test--node02/wordpress285-<wbr>data/html/wp-content/themes/<wbr>twentyfourteen/single.php</div><div>getfattr: Removing leading '/' from absolute path names</div><div># file: storage/sdc/brick_storage01/pr<wbr>ojects/183-57c559ea4d60e-canar<wbr>y-test--node02/wordpress285-<wbr>data/html/wp-content/themes/<wbr>twentyfourteen/single.php</div><div>security.selinux=0x73797374656<wbr>d5f753a6f626a6563745f723a676c7<wbr>573746572645f627269636b5f743a7<wbr>33000</div><div>trusted.afr.dirty=0x0000000000<wbr>00000000000000</div><div>trusted.bit-rot.version=0x0200<wbr>00000000000057868423000d6332</div><div>trusted.gfid=0x14f74b046793452<wbr>89dbd3290a3665cbc</div><div>trusted.glusterfs.quota.47e007<wbr>ee-6f91-4187-81f8-90a393deba2b<wbr>.contri.1=0x000000000000060000<wbr>00000000000001</div><div>trusted.pgfid.47e007ee-6f91-41<wbr>87-81f8-90a393deba2b=0x0000000<wbr>1</div></div><div><br></div><div><br></div><div><br></div><div>I can see the only the first brick has the appropiate trusted.afr.<client> tag - e.g in this case </div><div><br></div><div>trusted.afr.storage01-client-0<wbr>=0x000000020000000100000000<br></div><div><br></div><div>Files are same size under stat - just the access/modify/change dates are different.</div><div><br></div><div>My first question is - reading <a href="https://gluster.readthedocs.io/en/latest/Troubleshooting/split-brain/" target="_blank">https://gluster.readth<wbr>edocs.io/en/latest/Troubleshoo<wbr>ting/split-brain/</a> this suggests that i should have this field on both copies of the files - or am I mis-reading?</div><div><br></div><div>Secondly - am I correct that each one of these entries will require manual fixing? (I have approx 6K files/directories in this state over two clusters - which appears like an awful lot of manual fixing)</div><div><br></div><div>I've checked gluster volume info <vol> and all appropiate services/self-heal daemon are running. We've even tried a full heal/heal and iterating over parts of the filesystem in question with find / stat / md5sum.</div><div><br></div><div>Any input appreciated.</div><div><br></div><div>Cheers,</div><div><br></div><div><br></div><div><br></div><div><br></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></div></div>