[Gluster-users] [gluster] possible split-brain issue
Pranith Kumar Karampuri
pkarampu at redhat.com
Wed Jan 21 16:13:44 UTC 2015
On 01/14/2015 04:48 PM, Jifeng Li wrote:
>
> Hi ,
>
> [issue]: To ensure the glusterFS mount point work, a script will
> periodically using HTTP put a file to subdirectory under mount point
> which is used as Apache DocumentRoot. But after running some time,
> some errors show below:
>
> [2015-01-14 09:18:40.915639] E
> [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
> 0-gv0-replicate-0: metadata self heal failed, on /mpdis
>
> [2015-01-14 09:18:41.924584] E
> [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
> 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible
> split-brain). Please delete the file from all but the preferred
> subvolume.- Pending matrix: [ [ 0 20 ] [ 21 0 ] ]
>
> [2015-01-14 09:18:41.925182] E
> [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
> 0-gv0-replicate-0: metadata self heal failed, on /
>
> [2015-01-14 09:18:41.934827] E
> [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
> 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible
> split-brain). Please delete the file from all but the preferred
> subvolume.- Pending matrix: [ [ 0 4 ] [ 2 0 ] ]
>
> [2015-01-14 09:18:41.935375] E
> [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
> 0-gv0-replicate-0: metadata self heal failed, on /mpdis
>
> [2015-01-14 09:18:42.943742] E
> [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
> 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible
> split-brain). Please delete the file from all but the preferred
> subvolume.- Pending matrix: [ [ 0 20 ] [ 21 0 ] ]
>
> [2015-01-14 09:18:42.944432] E
> [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
> 0-gv0-replicate-0: metadata self heal failed, on /
>
> [2015-01-14 09:18:42.946664] E
> [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
> 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible
> split-brain). Please delete the file from all but the preferred
> subvolume.- Pending matrix: [ [ 0 4 ] [ 2 0 ] ]
>
> [2015-01-14 09:18:42.947323] E
> [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
> 0-gv0-replicate-0: metadata self heal failed, on /mpdis
>
> [2015-01-14 09:18:43.955929] E
> [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
> 0-gv0-replicate-0: Unable to self-heal contents of '/' (possible
> split-brain). Please delete the file from all but the preferred
> subvolume.- Pending matrix: [ [ 0 20 ] [ 21 0 ] ]
>
> [2015-01-14 09:18:43.956701] E
> [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
> 0-gv0-replicate-0: metadata self heal failed, on /
>
> [2015-01-14 09:18:43.958874] E
> [afr-self-heal-common.c:233:afr_sh_print_split_brain_log]
> 0-gv0-replicate-0: Unable to self-heal contents of '/mpdis' (possible
> split-brain). Please delete the file from all but the preferred
> subvolume.- Pending matrix: [ [ 0 4 ] [ 2 0 ] ]
>
> Besides, I find Input/output error shown below when listing the
> files of under mount point:
>
> [root at dmf-wpst-2 mpdis]# ll
>
> total 0
>
> -rwxr-xr-x. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf1
>
> -rw-r--r--. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf2
>
> [root at dmf-wpst-2 mpdis]# ll
>
> total 0
>
> -rwxr-xr-x. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf1
>
> -rw-r--r--. 1 apache apache 0 Jan 14 04:21 test.rep.00.00.00.00.dmf2
>
> [root at dmf-wpst-2 mpdis]# ll
>
> ls: cannot open directory .: Input/output error
>
> [root at dmf-wpst-2 mpdis]# ll
>
> ls: cannot access test.rep.00.00.00.00.dmf1: Input/output error
>
> ls: cannot access test.rep.00.00.00.00.dmf2: Input/output error
>
> total 0
>
> ?????????? ? ? ? ? ? test.rep.00.00.00.00.dmf1
>
> ?????????? ? ? ? ? ? test.rep.00.00.00.00.dmf2
>
> * Any tips about debugging further or getting this fixed up would
> be appreciated. *
>
> [version]: 3.5.3
>
> [environment]: two virtual server each has one brick :
>
> root at dmf-wpst-2 mpdis]# gluster volume status
>
> Status of volume: gv0
>
> Gluster process Port Online Pid
>
> ------------------------------------------------------------------------------
>
> Brick dmf-ha-1-glusterfs:/export/vdb1/brick 49152 Y 332
>
> Brick dmf-ha-2-glusterfs:/export/vdb1/brick 49154 Y 19396
>
> Self-heal Daemon on localhost N/A Y 19410
>
> Self-heal Daemon on 10.175.123.246 N/A Y 999
>
> [root at dmf-wpst-1 mpdis]# gluster volume info
>
> Volume Name: gv0
>
> Type: Replicate
>
> Volume ID: 51de44c3-f01e-486d-a6b7-10c7b7a270d7
>
> Status: Started
>
> Number of Bricks: 1 x 2 = 2
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: dmf-ha-1-glusterfs:/export/vdb1/brick
>
> Brick2: dmf-ha-2-glusterfs:/export/vdb1/brick
>
> Options Reconfigured:
>
> nfs.disable: ON
>
> network.ping-timeout: 2
>
> storage.bd-aio: on
>
> storage.linux-aio: on
>
> cluster.eager-lock: on
>
> performance.client-io-threads: on
>
> performance.cache-refresh-timeout: 60
>
> performance.io-thread-count: 64
>
> performance.cache-size: 8GB
>
> cluster.server-quorum-type: none
>
> [mount-point info]:
>
> 1.mount command
>
> glusterfs -p /var/run/glusterfs.pid
> --volfile-server=dmf-ha-1-glusterfs
> --volfile-server=dmf-ha-2-glusterfs --volfile-id=gv0 /dmfcontents
>
> 2.mount point directory hierarchy
>
> [root at dmf-wpst-2 /]# ls -ld /dmfcontents/
>
> drwxr-xr-x. 5 root root 71 Jan 14 04:39 /dmfcontents/
>
> [root at dmf-wpst-2 /]# ls -ld /dmfcontents/mpdis/
>
> drwxr-xr-x. 2 apache apache 89 Jan 14 04:39
> /dmfcontents/mpdis/
>
hi Jifeng Li,
Sorry for the delay in response. Could you post the output of:
'getfattr -d -m. -e hex <brick-path>'
'getfattr -d -m. -e hex <brick-path>/mpdis'
'getfattr -d -m. -e hex <brick-path>/mpdis/test.rep.00.00.00.00.dmf1'
'getfattr -d -m. -e hex <brick-path>/mpdis/test.rep.00.00.00.00.dmf2'
Pranith
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150121/26710104/attachment.html>
More information about the Gluster-users
mailing list