[Bugs] [Bug 1366849] New: add reporting on why files/folders are in split-brain

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1366849

            Bug ID: 1366849
           Summary: add reporting on why files/folders are in split-brain
           Product: GlusterFS
           Version: 3.7.14
         Component: selfheal
          Assignee: bugs at gluster.org
          Reporter: jaco at uls.co.za
                CC: bugs at gluster.org



Description of problem:

Feature request really.  As per IRC discussion:

<jkroon> a useful adition would be to be able to query WHY it things the file
is is split brain, eg: volume heal ${volname} info file <gfid:xxxxx>

Basically, given a path or gfid that's in split-brain, a way to dump the
information required to make an informed decision on what to do would be
extremely helpful.

A slightly improved way to resolve a gfid split (as what I've seen last night)
where a file path has different gfid values on different bricks would be most
welcome (the way I used - described lower down - is likely to be very error
prone unless you're particularly careful).

This follows a rather long discussion regarding split brains a few hours ago. 
full discussion below.

<jkroon> JoeJulian, whilst i've got you - i've read your blog on the whole gfid
thing a few times now, and from the output of volume heal ${volname} info
split-brain I'm still a little baffled as to how to get those resolved gfid
ones sorted out in a sensible manner.
<jkroon> mostly what I do is I run a find on one of the underlying bricks with
(assuming pwd is the brick folder) find . -samefile .gluster/aa/bb/fullgfid ...
which then gives me the path.
<jkroon> I think that assumes it's a file and not a folder ?
<jkroon> but with slightly over 5m files on the brick that can take a VERY long
time.
<jkroon> I'm guessing I can just compare meta-data on the gfid file itself
already ... but in case where I randomly shoot one of the two files (in the
cases I've had it's mostly log files so both are really wrong)
<JoeJulian> except folders are symlinks in the .glusterfs tree so you can't
really check the meta-data.
<jkroon> but also - in order to rm one of the two, you need all hard links to
it, so my current concede rm's both the gfid file and the other file - last
time I needed that was with 3.5
<jkroon> symlinks to the actual folders?
<jkroon> doesn't that mean we can use readlink -f to get to the real path?
<JoeJulian> There's really no valid reason for folders to go split-brain. iirc,
there's an open bug about them (unless it was fixed recently)
<jkroon> folders  that goes into split brain I've only ever seen for two
reasons:  1.  permissions differ on the bricks; 2.  time-stamps differ.
<JoeJulian> I've seen it for no reason at all.
* ChanServ gives channel operator status to hagarth
<JoeJulian> Just there was a trusted.afr attribute on both replica stating the
other one was out of date, but they completely matched.
<jkroon> theoretically ANY meta data differences can cause it.  i also read
something about files being modified could trigger it - but newer versions
handles that specific cse.
<jkroon> i've got 19 splits at the moment on one cluster, those that are not
gfids are all folders.
<JoeJulian> I fix folder splits by removing all trusted.afr from all
directories on the bricks.
<JoeJulian> @split-brain
<glusterbot> JoeJulian: (#1) To heal split brain, rtfm at
http://gluster.readthedocs.io/en/latest/Troubleshooting/heal-info-and-split-brain-resolution/
first., or (#2) For versions older than 3.7, see splitmount
https://joejulian.name/blog/glusterfs-split-brain-recovery-made-easy/, or (#3)
For manual brick-level recovery see
http://gluster.readthedocs.io/en/latest/Troubleshooting/split-brain/
<jkroon> JoeJulian, that first link actually perhaps explains what you're
seeing.
<JoeJulian> Of course even the manual repair documentation is wrong for
directories.
<jkroon> the folder goes into split-brain due to file in folder being in gfid
split brain.
<JoeJulian> Not in any of the cases I've looked at.
<JoeJulian> I have seen gfid splits, but that usually looks really ugly,
multiple copies of filenames and stuff.
<jkroon> a useful adition would be to be able to query WHY it things the file
is is split brain, eg: volume heal ${volname} info file <gfid:xxxxx>
<JoeJulian> jkroon: good idea, file a bug report
<glusterbot> https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS
<jkroon> JoeJulian, if the "volume ${volname} split-brain bigger-file
/path/to/file" where file is not listed as being in split brain, but is the
cause of the folder split brain refuses to heal - i'm guessing it's manual
repair time by removing it straight from the brick?
<jkroon> is it possible / why would the same file path map to two different
gfid values on different bricks?
* plarsen (~plarsen at redhat/jboss/pdpc.professional.plarsen) has joined #gluster
<JoeJulian> I haven't had a chance to try the setfattr method of repairing
split-brain on a directory yet.
<JoeJulian> Brick A is up (no quorum) and you create directory "foo". Brick "A"
goes down and brick "B" comes up. "foo" doesn't exist so you create it. Now
it's been created on two different bricks and each has been assigned a new
gfid.
<jkroon> oh boy.
<jkroon> ok, in my case now it's a file, but yes, that's a very viable scenario
seeing that I had a junior tech that wreacked havoc on that system in
particular.
<jkroon> yea, they're all a bunch of courierpop3dsizelist files inside folders
that's causing problems.
<jkroon> can't heal them because they have different gfid values.
<jkroon> and can't rm them via the mountpoint.

After this I manually tracked the problem-causing files using readlink -f
.gluster/.../${gfid} as reported by volume heal ... info split-brain and then
using ls on the mountpoint to find the files causing IO errors, and then on
each brick used getfattr to get the two different gfid values, then I removed
(due to not caring about the contents of the files - they'd be recreated in all
cases) both the gfid file as well as it's associated file from the bricks. 
After this sanity returned.

Point being, a way to determine WHAT is the reason for a file being reported as
split-brain would be extremely useful.  In the above scenario it could simply
state something like "file xyz in /path has differing gfid values: on
server1:/brick/path=gfid, server2:/brick/path=gfid" type of thing, or if the
files has differering content, provide the mtimes and file sizes.  This should
make it much easier and faster to decide what to do, not to mention providing a
little more detail.  In the case of gfid values being supplied it's probably a
good idea if we can (within reasonable time frames) resolve that gfid to an
actual path.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.