[Gluster-users] after hard reboot, split-brain happened, but nothing showed in gluster voluem heal info command !

Zhou, Cynthia (NSB - CN/Hangzhou) cynthia.zhou at nokia-sbell.com
Wed Sep 27 04:12:31 UTC 2017


HI gluster experts,

I meet a tough problem about “split-brain” issue. Sometimes, after hard reboot, we will find some files in split-brain, however its parent directory or anything could be shown in command “gluster volume heal <volume-name> info”, also, no entry in .glusterfs/indices/xattrop directory, can you help to shed some lights on this issue? Thanks!



Following is some info from our env,

Checking from sn-0 cliet, nothing is shown in-split-brain!

[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
# gluster v heal services info
Brick sn-0:/mnt/bricks/services/brick/
Number of entries: 0

Brick sn-1:/mnt/bricks/services/brick/
Number of entries: 0

[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
# gluster v heal services info split-brain
Gathering list of split brain entries on volume services has been successful

Brick sn-0.local:/mnt/bricks/services/brick
Number of entries: 0

Brick sn-1.local:/mnt/bricks/services/brick
Number of entries: 0

[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
# ls -l /mnt/services/netserv/ethip/
ls: cannot access '/mnt/services/netserv/ethip/sn-2': Input/output error
ls: cannot access '/mnt/services/netserv/ethip/mn-1': Input/output error
total 3
-rw-r--r-- 1 root root 144 Sep 26 20:35 as-0
-rw-r--r-- 1 root root 144 Sep 26 20:35 as-1
-rw-r--r-- 1 root root 145 Sep 26 20:35 as-2
-rw-r--r-- 1 root root 237 Sep 26 20:36 mn-0
-????????? ? ?    ?      ?            ? mn-1
-rw-r--r-- 1 root root  73 Sep 26 20:35 sn-0
-rw-r--r-- 1 root root  73 Sep 26 20:35 sn-1
-????????? ? ?    ?      ?            ? sn-2
[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]

Checking from glusterfs server side, the gfid of mn-1 on sn-0 and sn-1 is different

[SN-0]
[root at sn-0:/mnt/bricks/services/brick/.glusterfs/53/a3]
# getfattr -m . -d -e hex /mnt/bricks/services/brick/netserv/ethip
getfattr: Removing leading '/' from absolute path names
# file: mnt/bricks/services/brick/netserv/ethip
trusted.gfid=0xee71d19ac0f84f60b11eb42a083644e4
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
# getfattr -m . -d -e hex mn-1
# file: mn-1
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.services-client-0=0x000000000000000000000000
trusted.afr.services-client-1=0x000000000000000000000000
trusted.gfid=0x53a33f437464475486f31c4e44d83afd
[root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
# stat mn-1
  File: mn-1
  Size: 237             Blocks: 16         IO Block: 4096   regular file
Device: fd51h/64849d    Inode: 2536        Links: 2
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-09-26 20:30:25.679000000 +0300
Modify: 2017-09-26 20:30:24.604000000 +0300
Change: 2017-09-26 20:30:24.610000000 +0300
 Birth: -
[root at sn-0:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]
# ls
xattrop-63f8bbcb-7fa6-4fc8-b721-675a05de0ab3
[root at sn-0:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]

[root at sn-0:/mnt/bricks/services/brick/.glusterfs/53/a3]
# ls
53a33f43-7464-4754-86f3-1c4e44d83afd
[root at sn-0:/mnt/bricks/services/brick/.glusterfs/53/a3]
# stat 53a33f43-7464-4754-86f3-1c4e44d83afd
  File: 53a33f43-7464-4754-86f3-1c4e44d83afd
  Size: 237             Blocks: 16         IO Block: 4096   regular file
Device: fd51h/64849d    Inode: 2536        Links: 2
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-09-26 20:30:25.679000000 +0300
Modify: 2017-09-26 20:30:24.604000000 +0300
Change: 2017-09-26 20:30:24.610000000 +0300
 Birth: -

#
[SN-1]

[root at sn-1:/mnt/bricks/services/brick/.glusterfs/f7/f1]
#  getfattr -m . -d -e hex /mnt/bricks/services/brick/netserv/ethip
getfattr: Removing leading '/' from absolute path names
# file: mnt/bricks/services/brick/netserv/ethip
trusted.gfid=0xee71d19ac0f84f60b11eb42a083644e4
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

[root at sn-1:/mnt/bricks/services/brick/.glusterfs/f7/f1]
#
[root at sn-1:/mnt/bricks/services/brick/netserv/ethip]
# getfattr -m . -d -e hex mn-1
# file: mn-1
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.services-client-0=0x000000000000000000000000
trusted.afr.services-client-1=0x000000000000000000000000
trusted.gfid=0xf7f10f980acc4041a015e48018571d4a

[root at sn-1:/mnt/bricks/services/brick/netserv/ethip]
# stat mn-1
  File: mn-1
  Size: 237             Blocks: 16         IO Block: 4096   regular file
Device: fd41h/64833d    Inode: 2608        Links: 2
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-09-26 20:31:48.231000000 +0300
Modify: 2017-09-26 20:31:46.872000000 +0300
Change: 2017-09-26 20:31:46.875000000 +0300
 Birth: -
[root at sn-1:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]
# ls
xattrop-240713ea-eda3-4914-a55d-7dd4aed724ed
[root at sn-1:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]

[root at sn-1:/mnt/bricks/services/brick/.glusterfs/f7/f1]
# stat f7f10f98-0acc-4041-a015-e48018571d4a
  File: f7f10f98-0acc-4041-a015-e48018571d4a
  Size: 237             Blocks: 16         IO Block: 4096   regular file
Device: fd41h/64833d    Inode: 2608        Links: 2
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2017-09-26 20:31:48.231000000 +0300
Modify: 2017-09-26 20:31:46.872000000 +0300
Change: 2017-09-26 20:31:46.875000000 +0300
 Birth: -


Best regards,
Cynthia (周琳)
MBB SM HETRAN SW3 MATRIX
Storage
Mobile: +86 (0)18657188311



Best regards,
Cynthia (周琳)
MBB SM HETRAN SW3 MATRIX
Storage
Mobile: +86 (0)18657188311



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170927/03c2e591/attachment.html>


More information about the Gluster-users mailing list