[Gluster-users] after hard reboot, split-brain happened, but nothing showed in gluster voluem heal info command !
Karthik Subrahmanya
ksubrahm at redhat.com
Thu Sep 28 06:02:17 UTC 2017
Hi,
To resolve the gfid split-brain you can follow the steps at [1].
Since we don't have the pending markers set on the files, it is not showing
in the heal info.
To debug this issue, need some more data from you. Could you provide these
things?
1. volume info
2. mount log
3. brick logs
4. shd log
May I also know which version of gluster you are running. From the info you
have provided it looks like an old version.
If it is, then it would be great if you can upgarde to one of the latest
supported release.
[1]
http://docs.gluster.org/en/latest/Troubleshooting/split-brain/#fixing-directory-entry-split-brain
Thanks & Regards,
Karthik
On Wed, Sep 27, 2017 at 9:42 AM, Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:
>
> HI gluster experts,
>
> I meet a tough problem about “split-brain” issue. Sometimes, after hard
> reboot, we will find some files in split-brain, however its parent
> directory or anything could be shown in command “gluster volume heal
> <volume-name> info”, also, no entry in .glusterfs/indices/xattrop
> directory, can you help to shed some lights on this issue? Thanks!
>
>
>
> Following is some info from our env,
>
> *Checking from sn-0 cliet, nothing is shown in-split-brain!*
>
> [root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
> # gluster v heal services info
> Brick sn-0:/mnt/bricks/services/brick/
> Number of entries: 0
>
> Brick sn-1:/mnt/bricks/services/brick/
> Number of entries: 0
>
> [root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
> [root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
> # gluster v heal services info split-brain
> Gathering list of split brain entries on volume services has been
> successful
>
> Brick sn-0.local:/mnt/bricks/services/brick
> Number of entries: 0
>
> Brick sn-1.local:/mnt/bricks/services/brick
> Number of entries: 0
>
> [root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
> # ls -l /mnt/services/netserv/ethip/
> ls: cannot access '/mnt/services/netserv/ethip/sn-2': Input/output error
> ls: cannot access '/mnt/services/netserv/ethip/mn-1': Input/output error
> total 3
> -rw-r--r-- 1 root root 144 Sep 26 20:35 as-0
> -rw-r--r-- 1 root root 144 Sep 26 20:35 as-1
> -rw-r--r-- 1 root root 145 Sep 26 20:35 as-2
> -rw-r--r-- 1 root root 237 Sep 26 20:36 mn-0
> -????????? ? ? ? ? ? mn-1
> -rw-r--r-- 1 root root 73 Sep 26 20:35 sn-0
> -rw-r--r-- 1 root root 73 Sep 26 20:35 sn-1
> -????????? ? ? ? ? ? sn-2
> [root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
>
> *Checking from glusterfs server side, the gfid of mn-1 on sn-0 and sn-1 is
> different*
>
> *[SN-0]*
> [root at sn-0:/mnt/bricks/services/brick/.glusterfs/53/a3]
> # getfattr -m . -d -e hex /mnt/bricks/services/brick/netserv/ethip
> getfattr: Removing leading '/' from absolute path names
> # file: mnt/bricks/services/brick/netserv/ethip
> trusted.gfid=0xee71d19ac0f84f60b11eb42a083644e4
> trusted.glusterfs.dht=0x000000010000000000000000ffffffff
>
> [root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
> # getfattr -m . -d -e hex mn-1
> # file: mn-1
> trusted.afr.dirty=0x000000000000000000000000
> trusted.afr.services-client-0=0x000000000000000000000000
> trusted.afr.services-client-1=0x000000000000000000000000
> trusted.gfid=0x53a33f437464475486f31c4e44d83afd
> [root at sn-0:/mnt/bricks/services/brick/netserv/ethip]
> # stat mn-1
> File: mn-1
> Size: 237 Blocks: 16 IO Block: 4096 regular file
> Device: fd51h/64849d Inode: 2536 Links: 2
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2017-09-26 20:30:25.679000000 +0300
> Modify: 2017-09-26 20:30:24.604000000 +0300
> Change: 2017-09-26 20:30:24.610000000 +0300
> Birth: -
> [root at sn-0:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]
> # ls
> xattrop-63f8bbcb-7fa6-4fc8-b721-675a05de0ab3
> [root at sn-0:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]
>
> [root at sn-0:/mnt/bricks/services/brick/.glusterfs/53/a3]
> # ls
> 53a33f43-7464-4754-86f3-1c4e44d83afd
> [root at sn-0:/mnt/bricks/services/brick/.glusterfs/53/a3]
> # stat 53a33f43-7464-4754-86f3-1c4e44d83afd
> File: 53a33f43-7464-4754-86f3-1c4e44d83afd
> Size: 237 Blocks: 16 IO Block: 4096 regular file
> Device: fd51h/64849d Inode: 2536 Links: 2
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2017-09-26 20:30:25.679000000 +0300
> Modify: 2017-09-26 20:30:24.604000000 +0300
> Change: 2017-09-26 20:30:24.610000000 +0300
> Birth: -
>
> #
> *[SN-1]*
>
> [root at sn-1:/mnt/bricks/services/brick/.glusterfs/f7/f1]
> # getfattr -m . -d -e hex /mnt/bricks/services/brick/netserv/ethip
> getfattr: Removing leading '/' from absolute path names
> # file: mnt/bricks/services/brick/netserv/ethip
> trusted.gfid=0xee71d19ac0f84f60b11eb42a083644e4
> trusted.glusterfs.dht=0x000000010000000000000000ffffffff
>
> [root at sn-1:/mnt/bricks/services/brick/.glusterfs/f7/f1]
> *#*
> [root at sn-1:/mnt/bricks/services/brick/netserv/ethip]
> # getfattr -m . -d -e hex mn-1
> # file: mn-1
> trusted.afr.dirty=0x000000000000000000000000
> trusted.afr.services-client-0=0x000000000000000000000000
> trusted.afr.services-client-1=0x000000000000000000000000
> trusted.gfid=0xf7f10f980acc4041a015e48018571d4a
>
> [root at sn-1:/mnt/bricks/services/brick/netserv/ethip]
> # stat mn-1
> File: mn-1
> Size: 237 Blocks: 16 IO Block: 4096 regular file
> Device: fd41h/64833d Inode: 2608 Links: 2
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2017-09-26 20:31:48.231000000 +0300
> Modify: 2017-09-26 20:31:46.872000000 +0300
> Change: 2017-09-26 20:31:46.875000000 +0300
> Birth: -
> [root at sn-1:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]
> # ls
> xattrop-240713ea-eda3-4914-a55d-7dd4aed724ed
> [root at sn-1:/mnt/bricks/services/brick/.glusterfs/indices/xattrop]
>
> [root at sn-1:/mnt/bricks/services/brick/.glusterfs/f7/f1]
> # stat f7f10f98-0acc-4041-a015-e48018571d4a
> File: f7f10f98-0acc-4041-a015-e48018571d4a
> Size: 237 Blocks: 16 IO Block: 4096 regular file
> Device: fd41h/64833d Inode: 2608 Links: 2
> Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2017-09-26 20:31:48.231000000 +0300
> Modify: 2017-09-26 20:31:46.872000000 +0300
> Change: 2017-09-26 20:31:46.875000000 +0300
> Birth: -
>
>
> Best regards,
> *Cynthia **（周琳）*
> MBB SM HETRAN SW3 MATRIX
> Storage
> Mobile: +86 (0)18657188311
>
>
>
> Best regards,
> *Cynthia **（周琳）*
> MBB SM HETRAN SW3 MATRIX
> Storage
> Mobile: +86 (0)18657188311
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170928/44ba1bc5/attachment.html>
More information about the Gluster-users
mailing list