[Gluster-devel] Failed to open as split brain seen, returning EIO

[Roman Borisenko]

Hi!

I have errors in my application caused by GlusterFS:

copy(/storage/2060000/2064000/2064000/2064011/dat): failed to open stream:
Input/output error
readfile(/storage/2060000/2064000/2064000/2064011/dat): failed to open
stream: Input/output error

$ cat /storage/2120000/2125000/2125800/2125844/25/dat
cat: /storage/2120000/2125000/2125800/2125844/25/dat: Input/output error

On client Instance:

$ tail /var/log/gluster.log
[2012-10-25 07:17:14.904162] W
[afr-self-heal-data.c:831:afr_lookup_select_read_child_by_txn_type]
0-storage-replicate-0: /2120000/2125000/2125800/2125844/25/game.dat:
Possible split-brain
[2012-10-25 07:17:14.904225] W
[afr-common.c:1226:afr_detect_self_heal_by_lookup_status]
0-storage-replicate-0: split brain detected during lookup of
/2120000/2125000/2125800/2125844/25/game.dat.
[2012-10-25 07:17:14.911705] E
[afr-self-heal-common.c:2156:afr_self_heal_completion_cbk]
0-storage-replicate-0: background  data gfid self-heal failed on
/2120000/2125000/2125800/2125844/25/game.dat
[2012-10-25 07:17:14.911885] W [afr-open.c:213:afr_open]
0-storage-replicate-0: failed to open as split brain seen, returning EIO
[2012-10-25 07:17:14.912047] W [fuse-bridge.c:713:fuse_fd_cbk]
0-glusterfs-fuse: 457439: OPEN()
/2120000/2125000/2125800/2125844/25/game.dat => -1 (Input/output error)
[2012-10-25 07:17:22.386426] W
[afr-self-heal-data.c:831:afr_lookup_select_read_child_by_txn_type]
0-storage-replicate-0: /2120000/2125000/2125800/2125844/25/game.dat:
Possible split-brain
[2012-10-25 07:17:22.386469] W
[afr-common.c:1226:afr_detect_self_heal_by_lookup_status]
0-storage-replicate-0: split brain detected during lookup of
/2120000/2125000/2125800/2125844/25/game.dat.
[2012-10-25 07:17:22.393897] E
[afr-self-heal-common.c:2156:afr_self_heal_completion_cbk]
0-storage-replicate-0: background  data gfid self-heal failed on
/2120000/2125000/2125800/2125844/25/game.dat
[2012-10-25 07:17:22.397818] W [afr-open.c:213:afr_open]
0-storage-replicate-0: failed to open as split brain seen, returning EIO
[2012-10-25 07:17:22.397853] W [fuse-bridge.c:713:fuse_fd_cbk]
0-glusterfs-fuse: 459320: OPEN()
/2120000/2125000/2125800/2125844/25/game.dat => -1 (Input/output error)

$ glusterd -V
glusterfs 3.3.0 built on Jun  4 2012 11:39:45

$ gluster volume info
 Volume Name: storage
Type: Distributed-Replicate
Volume ID: a278889e-16fe-4913-9af6-eca4c099c484
Status: Started
Number of Bricks: 4 x 2 = 8
Transport-type: tcp
Bricks:
Brick1: gluster-1a:/storage/1a
Brick2: gluster-1b:/storage/1b
Brick3: gluster-2a:/storage/2a
Brick4: gluster-2b:/storage/2b
Brick5: gluster-1a:/storage/3a
Brick6: gluster-1b:/storage/3b
Brick7: gluster-2a:/storage/4a
Brick8: gluster-2b:/storage/4b
Options Reconfigured:
performance.stat-prefetch: off
auth.allow: 10.*

$ gluster volume status
Status of volume: storage
Gluster process Port Online Pid
------------------------------------------------------------------------------
Brick gluster-1a:/storage/1a 24009 Y 17302
Brick gluster-1b:/storage/1b 24009 Y 12188
Brick gluster-2a:/storage/2a 24009 Y 10863
Brick gluster-2b:/storage/2b 24009 Y 13486
Brick gluster-1a:/storage/3a 24011 Y 17308
Brick gluster-1b:/storage/3b 24011 Y 12194
Brick gluster-2a:/storage/4a 24011 Y 10869
Brick gluster-2b:/storage/4b 24011 Y 13492
NFS Server on localhost 38467 Y 17314
Self-heal Daemon on localhost N/A Y 17320
NFS Server on gluster-2a 38467 Y 10879
Self-heal Daemon on gluster-2a N/A Y 10885
NFS Server on gluster-1b 38467 Y 12200
Self-heal Daemon on gluster-1b N/A Y 12206
NFS Server on gluster-2b 38467 Y 13503
Self-heal Daemon on gluster-2b N/A Y 13509


How can I fix the situation?

--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20121025/dc265e97/attachment-0001.html>