[Gluster-devel] after hard reboot, split-brain happened, but nothing showed in gluster voluem heal info command !
Zhou, Cynthia (NSB - CN/Hangzhou)
cynthia.zhou at nokia-sbell.com
Wed Sep 27 04:12:31 UTC 2017
HI gluster experts,
I meet a tough problem about “split-brain” issue. Sometimes, after hard reboot, we will find some files in split-brain, however its parent directory or anything could be shown in command “gluster volume heal <volume-name> info”, also, no entry in .glusterfs/indices/xattrop directory, can you help to shed some lights on this issue? Thanks!
Following is some info from our env,
Checking from sn-0 cliet, nothing is shown in-split-brain!
[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
# gluster v heal services info
Brick sn-0:/mnt/bricks/services/brick/
Number of entries: 0
Brick sn-1:/mnt/bricks/services/brick/
Number of entries: 0
[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
# gluster v heal services info split-brain
Gathering list of split brain entries on volume services has been successful
Brick sn-0.local:/mnt/bricks/services/brick
Number of entries: 0
Brick sn-1.local:/mnt/bricks/services/brick
Number of entries: 0
[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
# ls -l /mnt/services/netserv/ethip/
ls: cannot access '/mnt/services/netserv/ethip/sn-2': Input/output error
ls: cannot access '/mnt/services/netserv/ethip/mn-1': Input/output error
total 3
-rw-r--r-- 1 root root 144 Sep 26 20:35 as-0
-rw-r--r-- 1 root root 144 Sep 26 20:35 as-1
-rw-r--r-- 1 root root 145 Sep 26 20:35 as-2
-rw-r--r-- 1 root root 237 Sep 26 20:36 mn-0
-????????? ? ? ? ? ? mn-1
-rw-r--r-- 1 root root 73 Sep 26 20:35 sn-0
-rw-r--r-- 1 root root 73 Sep 26 20:35 sn-1
-????????? ? ? ? ? ? sn-2
[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
Checking from glusterfs server side, the gfid of mn-1 on sn-0 and sn-1 is different
[SN-0]
[root at sn-0:/mnt/bricks/services/brick/.glusterfs/53/a3]
# getfattr -m . -d -e hex /mnt/bricks/services/brick/netserv/ethip
getfattr: Removing leading '/' from absolute path names
# file: mnt/bricks/services/brick/netserv/ethip
trusted.gfid=0xee71d19ac0f84f60b11eb42a083644e4
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
# getfattr -m . -d -e hex mn-1
# file: mn-1
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.services-client-0=0x000000000000000000000000
trusted.afr.services-client-1=0x000000000000000000000000
trusted.gfid=0x53a33f437464475486f31c4e44d83afd
[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
# stat mn-1
File: mn-1
Size: 237 Blocks: 16 IO Block: 4096 regular file
Device: fd51h/64849d Inode: 2536 Links: 2
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2017-09-26 20:30:25.679000000 +0300
Modify: 2017-09-26 20:30:24.604000000 +0300
Change: 2017-09-26 20:30:24.610000000 +0300
Birth: -
[root at sn-0:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]
# ls
xattrop-63f8bbcb-7fa6-4fc8-b721-675a05de0ab3
[root at sn-0:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]
[root at sn-0:/mnt/bricks/services/brick/.glusterfs/53/a3]
# ls
53a33f43-7464-4754-86f3-1c4e44d83afd
[root at sn-0:/mnt/bricks/services/brick/.glusterfs/53/a3]
# stat 53a33f43-7464-4754-86f3-1c4e44d83afd
File: 53a33f43-7464-4754-86f3-1c4e44d83afd
Size: 237 Blocks: 16 IO Block: 4096 regular file
Device: fd51h/64849d Inode: 2536 Links: 2
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2017-09-26 20:30:25.679000000 +0300
Modify: 2017-09-26 20:30:24.604000000 +0300
Change: 2017-09-26 20:30:24.610000000 +0300
Birth: -
#
[SN-1]
[root at sn-1:/mnt/bricks/services/brick/.glusterfs/f7/f1]
# getfattr -m . -d -e hex /mnt/bricks/services/brick/netserv/ethip
getfattr: Removing leading '/' from absolute path names
# file: mnt/bricks/services/brick/netserv/ethip
trusted.gfid=0xee71d19ac0f84f60b11eb42a083644e4
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
[root at sn-1:/mnt/bricks/services/brick/.glusterfs/f7/f1]
#
[root at sn-1:/mnt/bricks/services/brick/netserv/ethip]
# getfattr -m . -d -e hex mn-1
# file: mn-1
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.services-client-0=0x000000000000000000000000
trusted.afr.services-client-1=0x000000000000000000000000
trusted.gfid=0xf7f10f980acc4041a015e48018571d4a
[root at sn-1:/mnt/bricks/services/brick/netserv/ethip]
# stat mn-1
File: mn-1
Size: 237 Blocks: 16 IO Block: 4096 regular file
Device: fd41h/64833d Inode: 2608 Links: 2
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2017-09-26 20:31:48.231000000 +0300
Modify: 2017-09-26 20:31:46.872000000 +0300
Change: 2017-09-26 20:31:46.875000000 +0300
Birth: -
[root at sn-1:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]
# ls
xattrop-240713ea-eda3-4914-a55d-7dd4aed724ed
[root at sn-1:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]
[root at sn-1:/mnt/bricks/services/brick/.glusterfs/f7/f1]
# stat f7f10f98-0acc-4041-a015-e48018571d4a
File: f7f10f98-0acc-4041-a015-e48018571d4a
Size: 237 Blocks: 16 IO Block: 4096 regular file
Device: fd41h/64833d Inode: 2608 Links: 2
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2017-09-26 20:31:48.231000000 +0300
Modify: 2017-09-26 20:31:46.872000000 +0300
Change: 2017-09-26 20:31:46.875000000 +0300
Birth: -
Best regards,
Cynthia （周琳）
MBB SM HETRAN SW3 MATRIX
Storage
Mobile: +86 (0)18657188311
Best regards,
Cynthia （周琳）
MBB SM HETRAN SW3 MATRIX
Storage
Mobile: +86 (0)18657188311
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20170927/03c2e591/attachment-0001.html>
More information about the Gluster-devel
mailing list