[Gluster-users] split-brain.. okay, what next?
Ravishankar N
ravishankar at redhat.com
Thu Jul 16 02:03:21 UTC 2015
On 07/16/2015 01:28 AM, Игорь Бирюлин wrote:
> I have studied information on page:
> https://github.com/gluster/glusterfs/blob/master/doc/features/heal-info-and-split-brain-resolution.md
> and cannot solve split-brain by this instruction.
>
> I have tested it on gluster 3.6 and it doesn't work, only on gluster 3.7.
>
Right. We need to explicitly mention in the .md that it is supported
from 3.7 onwards.
> I try to use on gluster 3.7.2.
> I have a gluster share in replicate mode:
> root at dist-gl2:/# gluster volume info
>
> Volume Name: repofiles
> Type: Replicate
> Volume ID: 1d5d5d7d-39f2-4011-9fc8-d73c29495e7c
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: dist-gl1:/brick1
> Brick2: dist-gl2:/brick1
> Options Reconfigured:
> performance.readdir-ahead: on
> server.allow-insecure: on
> root at dist-gl2:/#
>
> And I have one file in split-brain (it is file "test"):
> root at dist-gl2:/# gluster volume heal repofiles info
> Brick dist-gl1:/brick1/
> /test
> / - Is in split-brain
>
> Number of entries: 2
>
> Brick dist-gl2:/brick1/
> / - Is in split-brain
>
> /test
> Number of entries: 2
>
> root at dist-gl2:/# gluster volume heal repofiles info split-brain
> Brick dist-gl1:/brick1/
> /
> Number of entries in split-brain: 1
>
> Brick dist-gl2:/brick1/
> /
> Number of entries in split-brain: 1
>
> root at dist-gl2:/#
>
> I don't know why these commands show only directory ("/") in split-brain.
That is because the file is in gfid split-brain. As listed in the .md
file, " for a gfid split-brain, the parent directory of the file is
shown to be in split-brain and the file itself is shown to be needing
heal". You cannot resolve gfid split-brains using the commands. You need
to resolve them manually. See "Fixing Directory entry split-brain" in
https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
>
> I try solve split-brain by gluster cli commands (on directory from the
> output previous commands and on file), but it could not help:
> root at dist-gl2:/# gluster v heal repofiles split-brain bigger-file /
> Healing / failed:Operation not permitted.
> Volume heal failed.
> root at dist-gl2:/# gluster v heal repofiles split-brain bigger-file /test
> Lookup failed on /test:Input/output error
> Volume heal failed.
> root at dist-gl2:/# gluster v heal repofiles split-brain source-brick
> dist-gl1:/brick1 /
> Healing / failed:Operation not permitted.
> Volume heal failed.
> root at dist-gl2:/# gluster v heal repofiles split-brain source-brick
> dist-gl1:/brick1 /test
> Lookup failed on /test:Input/output error
> Volume heal failed.
> root at dist-gl2:/# gluster v heal repofiles split-brain source-brick
> dist-gl2:/brick1 /
> Healing / failed:Operation not permitted.
> Volume heal failed.
> root at dist-gl2:/# gluster v heal repofiles split-brain source-brick
> dist-gl2:/brick1 /test
> Lookup failed on /test:Input/output error
> Volume heal failed.
> root at dist-gl2:/#
>
> Parts of glfsheal-repofiles.log logs.
> When try to solve split-brain on dirictory ("/"):
> [2015-07-15 19:45:30.508670] I
> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started
> thread with index 1
> [2015-07-15 19:45:30.516662] I
> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started
> thread with index 2
> [2015-07-15 19:45:30.517201] I [MSGID: 104045]
> [glfs-master.c:95:notify] 0-gfapi: New graph
> 64697374-2d67-6c32-2d32-303634362d32 (0) coming up
> [2015-07-15 19:45:30.517227] I [MSGID: 114020] [client.c:2118:notify]
> 0-repofiles-client-0: parent translators are ready, attempting connect
> on transport
> [2015-07-15 19:45:30.525457] I [MSGID: 114020] [client.c:2118:notify]
> 0-repofiles-client-1: parent translators are ready, attempting connect
> on transport
> [2015-07-15 19:45:30.526788] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
> 0-repofiles-client-0: changing port to 49152 (from 0)
> [2015-07-15 19:45:30.534012] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
> 0-repofiles-client-1: changing port to 49152 (from 0)
> [2015-07-15 19:45:30.536252] I [MSGID: 114057]
> [client-handshake.c:1438:select_server_supported_programs]
> 0-repofiles-client-0: Using Program GlusterFS 3.3, Num (1298437),
> Version (330)
> [2015-07-15 19:45:30.536606] I [MSGID: 114046]
> [client-handshake.c:1214:client_setvolume_cbk] 0-repofiles-client-0:
> Connected to repofiles-client-0, attached to remote volume '/brick1'.
> [2015-07-15 19:45:30.536621] I [MSGID: 114047]
> [client-handshake.c:1225:client_setvolume_cbk] 0-repofiles-client-0:
> Server and Client lk-version numbers are not same, reopening the fds
> [2015-07-15 19:45:30.536679] I [MSGID: 108005]
> [afr-common.c:3883:afr_notify] 0-repofiles-replicate-0: Subvolume
> 'repofiles-client-0' came back up; going online.
> [2015-07-15 19:45:30.536819] I [MSGID: 114035]
> [client-handshake.c:193:client_set_lk_version_cbk]
> 0-repofiles-client-0: Server lk version = 1
> [2015-07-15 19:45:30.543712] I [MSGID: 114057]
> [client-handshake.c:1438:select_server_supported_programs]
> 0-repofiles-client-1: Using Program GlusterFS 3.3, Num (1298437),
> Version (330)
> [2015-07-15 19:45:30.543919] I [MSGID: 114046]
> [client-handshake.c:1214:client_setvolume_cbk] 0-repofiles-client-1:
> Connected to repofiles-client-1, attached to remote volume '/brick1'.
> [2015-07-15 19:45:30.543933] I [MSGID: 114047]
> [client-handshake.c:1225:client_setvolume_cbk] 0-repofiles-client-1:
> Server and Client lk-version numbers are not same, reopening the fds
> [2015-07-15 19:45:30.554650] I [MSGID: 114035]
> [client-handshake.c:193:client_set_lk_version_cbk]
> 0-repofiles-client-1: Server lk version = 1
> [2015-07-15 19:45:30.557628] I
> [afr-self-heal-entry.c:565:afr_selfheal_entry_do]
> 0-repofiles-replicate-0: performing entry selfheal on
> 00000000-0000-0000-0000-000000000001
> [2015-07-15 19:45:30.560002] E
> [afr-self-heal-entry.c:249:afr_selfheal_detect_gfid_and_type_mismatch]
> 0-repofiles-replicate-0: Gfid mismatch detected for
> <00000000-0000-0000-0000-000000000001/test>,
> e42d3f03-0633-4954-95ce-5cd8710e595e on repofiles-client-1 and
> 16da3178-8a6e-4010-b874-7f11449d1993 on repofiles-client-0. Skipping
> conservative merge on the file.
> [2015-07-15 19:45:30.561582] W [afr-common.c:1985:afr_discover_done]
> 0-repofiles-replicate-0: no read subvols for /
> [2015-07-15 19:45:30.561604] I
> [afr-common.c:1673:afr_local_discovery_cbk] 0-repofiles-replicate-0:
> selecting local read_child repofiles-client-1
> [2015-07-15 19:45:30.561900] W [afr-common.c:1985:afr_discover_done]
> 0-repofiles-replicate-0: no read subvols for /
> [2015-07-15 19:45:30.561962] I [MSGID: 104041]
> [glfs-resolve.c:843:__glfs_active_subvol] 0-repofiles: switched to
> graph 64697374-2d67-6c32-2d32-303634362d32 (0)
> [2015-07-15 19:45:30.562259] W [afr-common.c:1985:afr_discover_done]
> 0-repofiles-replicate-0: no read subvols for /
> [2015-07-15 19:45:32.563285] W [afr-common.c:1985:afr_discover_done]
> 0-repofiles-replicate-0: no read subvols for /
> [2015-07-15 19:45:32.564898] I
> [afr-self-heal-entry.c:565:afr_selfheal_entry_do]
> 0-repofiles-replicate-0: performing entry selfheal on
> 00000000-0000-0000-0000-000000000001
> [2015-07-15 19:45:32.566693] E
> [afr-self-heal-entry.c:249:afr_selfheal_detect_gfid_and_type_mismatch]
> 0-repofiles-replicate-0: Gfid mismatch detected for
> <00000000-0000-0000-0000-000000000001/test>,
> e42d3f03-0633-4954-95ce-5cd8710e595e on repofiles-client-1 and
> 16da3178-8a6e-4010-b874-7f11449d1993 on repofiles-client-0. Skipping
> conservative merge on the file.
> When try to solve split-brain on file ("/test"):
> [2015-07-15 19:48:45.910819] I
> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started
> thread with index 1
> [2015-07-15 19:48:45.919854] I
> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started
> thread with index 2
> [2015-07-15 19:48:45.920434] I [MSGID: 104045]
> [glfs-master.c:95:notify] 0-gfapi: New graph
> 64697374-2d67-6c32-2d32-313133392d32 (0) coming up
> [2015-07-15 19:48:45.920481] I [MSGID: 114020] [client.c:2118:notify]
> 0-repofiles-client-0: parent translators are ready, attempting connect
> on transport
> [2015-07-15 19:48:45.996442] I [MSGID: 114020] [client.c:2118:notify]
> 0-repofiles-client-1: parent translators are ready, attempting connect
> on transport
> [2015-07-15 19:48:45.997892] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
> 0-repofiles-client-0: changing port to 49152 (from 0)
> [2015-07-15 19:48:46.005153] I [rpc-clnt.c:1819:rpc_clnt_reconfig]
> 0-repofiles-client-1: changing port to 49152 (from 0)
> [2015-07-15 19:48:46.007437] I [MSGID: 114057]
> [client-handshake.c:1438:select_server_supported_programs]
> 0-repofiles-client-0: Using Program GlusterFS 3.3, Num (1298437),
> Version (330)
> [2015-07-15 19:48:46.007928] I [MSGID: 114046]
> [client-handshake.c:1214:client_setvolume_cbk] 0-repofiles-client-0:
> Connected to repofiles-client-0, attached to remote volume '/brick1'.
> [2015-07-15 19:48:46.007945] I [MSGID: 114047]
> [client-handshake.c:1225:client_setvolume_cbk] 0-repofiles-client-0:
> Server and Client lk-version numbers are not same, reopening the fds
> [2015-07-15 19:48:46.008020] I [MSGID: 108005]
> [afr-common.c:3883:afr_notify] 0-repofiles-replicate-0: Subvolume
> 'repofiles-client-0' came back up; going online.
> [2015-07-15 19:48:46.008189] I [MSGID: 114035]
> [client-handshake.c:193:client_set_lk_version_cbk]
> 0-repofiles-client-0: Server lk version = 1
> [2015-07-15 19:48:46.014313] I [MSGID: 114057]
> [client-handshake.c:1438:select_server_supported_programs]
> 0-repofiles-client-1: Using Program GlusterFS 3.3, Num (1298437),
> Version (330)
> [2015-07-15 19:48:46.014536] I [MSGID: 114046]
> [client-handshake.c:1214:client_setvolume_cbk] 0-repofiles-client-1:
> Connected to repofiles-client-1, attached to remote volume '/brick1'.
> [2015-07-15 19:48:46.014550] I [MSGID: 114047]
> [client-handshake.c:1225:client_setvolume_cbk] 0-repofiles-client-1:
> Server and Client lk-version numbers are not same, reopening the fds
> [2015-07-15 19:48:46.026828] I [MSGID: 114035]
> [client-handshake.c:193:client_set_lk_version_cbk]
> 0-repofiles-client-1: Server lk version = 1
> [2015-07-15 19:48:46.029357] I
> [afr-self-heal-entry.c:565:afr_selfheal_entry_do]
> 0-repofiles-replicate-0: performing entry selfheal on
> 00000000-0000-0000-0000-000000000001
> [2015-07-15 19:48:46.031719] E
> [afr-self-heal-entry.c:249:afr_selfheal_detect_gfid_and_type_mismatch]
> 0-repofiles-replicate-0: Gfid mismatch detected for
> <00000000-0000-0000-0000-000000000001/test>,
> e42d3f03-0633-4954-95ce-5cd8710e595e on repofiles-client-1 and
> 16da3178-8a6e-4010-b874-7f11449d1993 on repofiles-client-0. Skipping
> conservative merge on the file.
> [2015-07-15 19:48:46.033222] W [afr-common.c:1985:afr_discover_done]
> 0-repofiles-replicate-0: no read subvols for /
> [2015-07-15 19:48:46.033224] I
> [afr-common.c:1673:afr_local_discovery_cbk] 0-repofiles-replicate-0:
> selecting local read_child repofiles-client-1
> [2015-07-15 19:48:46.033569] W [afr-common.c:1985:afr_discover_done]
> 0-repofiles-replicate-0: no read subvols for /
> [2015-07-15 19:48:46.033624] I [MSGID: 104041]
> [glfs-resolve.c:843:__glfs_active_subvol] 0-repofiles: switched to
> graph 64697374-2d67-6c32-2d32-313133392d32 (0)
> [2015-07-15 19:48:46.033906] W [afr-common.c:1985:afr_discover_done]
> 0-repofiles-replicate-0: no read subvols for /
> [2015-07-15 19:48:48.036482] W [MSGID: 108008]
> [afr-self-heal-name.c:353:afr_selfheal_name_gfid_mismatch_check]
> 0-repofiles-replicate-0: GFID mismatch for
> <gfid:00000000-0000-0000-0000-000000000001>/test
> e42d3f03-0633-4954-95ce-5cd8710e595e on repofiles-client-1 and
> 16da3178-8a6e-4010-b874-7f11449d1993 on repofiles-client-0
>
> Where I did mistake when try solve split-brain?
>
> Best regards,
> Igor
>
> 2015-07-14 22:11 GMT+03:00 Roman <romeo.r at gmail.com
> <mailto:romeo.r at gmail.com>>:
>
> never mind. I do not have enough time to debug why basic commands
> of gluster do not work on production server. It was enough of
> tonight's system freeze due to not documented XFS settings MUST
> have to run glusterfs with XFS. I'll keep to EXT4 better. Anyway
> XFS for bricks did not solve my previous problem.
>
> To solve split-brain this time, I've restored VM from backup.
>
> 2015-07-14 21:55 GMT+03:00 Roman <romeo.r at gmail.com
> <mailto:romeo.r at gmail.com>>:
>
> Thanx for pointing out...
> but doesn't seem to work... or i am too sleepy due to problems
> with glusterfs and debian8 in other topic which i'm fighting
> for month..
>
> root at stor1:~# gluster volume heal HA-2TB-TT-Proxmox-cluster
> split-brain source-brick stor1:HA-2TB-TT-Proxmox-cluster/2TB
> /images/124/vm-124-disk-1.qcow2
> Usage: volume heal <VOLNAME> [{full | statistics {heal-count
> {replica <hostname:brickname>}} |info {healed | heal-failed |
> split-brain}}]
>
> seems like wrong command...
>
> 2015-07-14 21:23 GMT+03:00 Joe Julian <joe at julianfamily.org
> <mailto:joe at julianfamily.org>>:
>
> On 07/14/2015 11:19 AM, Roman wrote:
>
> Hi,
>
> played with glusterfs tonight and tried to use
> recommended XFS for gluster.. first try was pretty bad
> and all of my VM-s hanged (XFS wants allocsize=64k to
> create qcow2 files, which i didn't know about and
> tried to create VM on XFS without this config line in
> fstab, which lead to a lot of IO-s and qemu says it
> got time out while creating the file)..
>
> now i've got this:
> Brick stor1:/exports/HA-2TB-TT-Proxmox-cluster/2TB/
> /images/124/vm-124-disk-1.qcow2 - Is in split-brain
>
> Number of entries: 1
>
> Brick stor2:/exports/HA-2TB-TT-Proxmox-cluster/2TB/
> /images/124/vm-124-disk-1.qcow2 - Is in split-brain
>
> ok. what next?
> I've deleted one of files, it didn't help. even more,
> selfheal restored the file on node, where i deleted
> it... and still split-brain.
>
> how to fix?
>
> --
> Best regards,
> Roman.
>
>
>
> https://github.com/gluster/glusterfs/blob/master/doc/features/heal-info-and-split-brain-resolution.md
>
>
> or
>
> https://joejulian.name/blog/glusterfs-split-brain-recovery-made-easy/
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
> Best regards,
> Roman.
>
>
>
>
> --
> Best regards,
> Roman.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150716/b5ba48cf/attachment.html>
More information about the Gluster-users
mailing list