[Gluster-users] gluster & split-brain mess

[Ravishankar N]

On 05/31/2016 08:19 PM, Johan Moreels wrote:
> Dear developers,
>
>
> I would like to thank you for the great job in developing gluster for the
> open-source community and I value the various improvements in the successive
> versions from 3.4 up to 3.7 that I used.
>
> However, as I already mentionned in my older posts, gluster has an issue to
> handle high loads, i.e. normal use, when multiple clients try to access its
> volume. This issue combined to the fact that in the past I used versions
> suffering from serious memory leaks (for example 3.6.9 which despite the fact
> that the issues were known has not been explicitly discouraged to use) which
> results in brick processes being killed by OOM kernel process. It results in
> about 40,000 files currently in split-brain conditions, while gluster was used
> normally.
>
> The current advise from the developers to solve the split-brain cases is
> to fix them manually which in my case is just not possible. This is why I plea
> that you urgently decide to include the CLI tools that allow to
> list according to the type of split-brain (either files diff and/or attributes
> diff and if attributes, what kind of attributes, i.e. quota) and a automatic
> way to fix a specific type of split-brain. As an example, if split-brain is
> detected due to mismatch between quota attributes for many files, the CLI
> should allow to invalidate these attributes and refresh them at once for all
> these files. As I already said, it is nice to add new features, but it is a
> little bit problematic if some basic management/repair tools are lacking.
>
> I must really stress that it is not the task of users to develop
> strategies/scripts to fix split-brain issues (additional amount of work to our
> normal work) but the developers, or am I missing something here ?
Hi Johan,

Recent versions of gluster (3.7.x, IIRC) have a CLI that you can use to 
fix split-brains based on various policies instead of messing around 
with the extended attributes yourself.
You can also resolve split-brains from the mount point using a virtual 
setfattr command. Both these methods are detailed in [1]. The soon to be 
released glusterfs 3.8
also has a new volume option [2] which can be set to automatically pick 
a source based on various policies.

>
> And finally, could you get rid of any gfid in the logs and heal CLI output but
> instead give the file or directory, because the gfid alone is simply useless
> to find the file/dir on multi-terabyte volume/bricks in a decent amount of
> time ?
We print the gfid only if the file name is not available from the brick 
process (because the brick just came back up and there was no lookup 
made on the brick from a client. i.e. the inode/dentry information is 
not yet populated in brick process).
The CLI referred to in [1] also accepts gfids to automate split-brain 
resolution. There are ways [3] to find the file name from gfid without 
crawling the filesystem for an inode number match.

Hope this helps,
Ravi

[1]https://github.com/gluster/glusterfs-specs/blob/master/done/Features/heal-info-and-split-brain-resolution.md
[2] 
http://review.gluster.org/#/c/14554/2/done/GlusterFS+3.8/automagic-unsplit-brain.md
[3] https://gluster.readthedocs.io/en/latest/Troubleshooting/gfid-to-path/


>
> Thanks,
>
>
> Alessandro.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users