[Gluster-users] gluster volume heal info split brain command not showing files in split-brain

Anuradha Talur atalur at redhat.com
Wed Mar 16 12:36:53 UTC 2016 


----- Original Message -----
> From: "Anuradha Talur" <atalur at redhat.com>
> To: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
> Cc: gluster-users at gluster.org, gluster-devel at gluster.org
> Sent: Wednesday, March 16, 2016 5:32:26 PM
> Subject: Re: [Gluster-users] gluster volume heal info split brain command not showing files in split-brain
> 
> 
> 
> ----- Original Message -----
> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
> > To: "Anuradha Talur" <atalur at redhat.com>
> > Cc: gluster-users at gluster.org, gluster-devel at gluster.org
> > Sent: Wednesday, March 16, 2016 4:39:26 PM
> > Subject: Re: [Gluster-users] gluster volume heal info split brain command
> > not showing files in split-brain
> > 
> > Hi Anuradha,
> > 
> > I am doing the same which is mentioned in the link you shared and It has
> > been resolved the issue.
> > 
> > But my question is if it is the split-brain scenario then why the command
> > "gluster volume heal info split-brain"
> >  not showing these files in the output even not the parent directory is
> > present in split-brain.
> > 
> > Please find the requested logs

Abhishek,

Yes, ideally it should show. I will look into it. The only reason I can think of, is when parent directory did not have any pending markers to indicate split-brain; which is why I asked getfattr output for the parent directory too. But if the issue is resolved, there isn't much info we can get out of it. Thanks for sharing the logs. Will see what could have caused this.

> Abhishek,
> 
> Yes, ideally it should show. I will look into it.
> I saw another case with this issue. The parent directory did not have any pe
> > 
> > On Wed, Mar 16, 2016 at 4:20 PM, Anuradha Talur <atalur at redhat.com> wrote:
> > 
> > > Hi Abhishek,
> > >
> > > The files that are reporting i/o error have gfid-mismatch. This situation
> > > is called directory or entry split-brain. You can find steps to resolve
> > > this kind of split brain here :
> > > https://gluster.readthedocs.org/en/latest/Troubleshooting/split-brain/ .
> > >
> > > Ideally, the parent directories of these files have to be listed in heal
> > > info split-brain output. Can you please get extended attributes of parent
> > > directories of the files that show i/o error (Same getfattr command that
> > > you previously used.) ?
> > >
> > > ----- Original Message -----
> > > > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
> > > > To: "Anuradha Talur" <atalur at redhat.com>
> > > > Cc: gluster-users at gluster.org, gluster-devel at gluster.org
> > > > Sent: Thursday, March 10, 2016 11:22:35 AM
> > > > Subject: Re: [Gluster-users] gluster volume heal info split brain
> > > command not showing files in split-brain
> > > >
> > > > Hi Anuradha,
> > > >
> > > > Please find the glusterfs and glusterd logs directory as an attachment.
> > > >
> > > > Regards,
> > > > Abhishek
> > > >
> > > >
> > > >
> > > > On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL <
> > > abhishpaliwal at gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Anuradha,
> > > > >
> > > > > Sorry for late reply.
> > > > >
> > > > > Please find the requested logs below:
> > > > >
> > > > > Remote: 10.32.0.48
> > > > > Local : 10.32.1.144
> > > > >
> > > > > Local:
> > > > > #gluster volume heal c_glusterfs info split-brain
> > > > > Brick 10.32.1.144:/opt/lvmdir/c2/brick
> > > > > Number of entries in split-brain: 0
> > > > >
> > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick
> > > > > Number of entries in split-brain: 0
> > > > >
> > > > > Remote:
> > > > > #gluster volume heal c_glusterfs info split-brain
> > > > > Brick 10.32.1.144:/opt/lvmdir/c2/brick
> > > > > Number of entries in split-brain: 0
> > > > >
> > > > > Brick 10.32.0.48:/opt/lvmdir/c2/brick
> > > > > Number of entries in split-brain: 0
> > > > >
> > > > > auto-sync.sh.
> > > > > Here you can see that i/o error is detected. Below is the required
> > > > > meta
> > > > > data from both the bricks.
> > > > >
> > > > > 1)
> > > > > stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior':
> > > > > Input/output error
> > > > > Remote:
> > > > >
> > > > > getfattr -d -m . -e hex
> > > > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > > > > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > > > > trusted.afr.dirty=0x000000000000000000000000
> > > > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f
> > > > > trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee
> > > > >
> > > > > Local:
> > > > >
> > > > > getfattr -d -m . -e hex
> > > > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > > > > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > > > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51
> > > > > trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd
> > > > >
> > > > >
> > > > > 2)
> > > > > stat: cannot stat '/mnt/c//security/corbasecurity': Input/output
> > > > > error
> > > > > Remote:
> > > > >
> > > > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
> > > > > # file: opt/lvmdir/c2/brick/security/corbasecurity
> > > > > trusted.afr.dirty=0x000000000000000000000000
> > > > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f
> > > > > trusted.gfid=0xd298b7a0c8834f3e99abb39741363013
> > > > >
> > > > > Local:
> > > > >
> > > > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
> > > > > # file: opt/lvmdir/c2/brick/security/corbasecurity
> > > > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51
> > > > > trusted.gfid=0x890df0f706184b52803fac3242a2f15b
> > > > >
> > > > > I observed that getfattr command output doesn't show all the fields
> > > > > all
> > > > > the times.
> > > > >
> > > > > Here you can check that gluster split-brain command hasn't reported
> > > > > any
> > > > > split-brains but resulted in IO errors when accessed few files.
> > > > >
> > > > > Could you please tell me if "split-brain" command doesn't reported
> > > > > any
> > > > > entry as output, then is there any way through which we can find out
> > > that
> > > > > the files are in split-brain if we are getting the IO error on those
> > > file.
> > > > >
> > > > >
> > > > > Regards,
> > > > > Abhishek
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur <atalur at redhat.com>
> > > wrote:
> > > > >
> > > > >>
> > > > >>
> > > > >> ----- Original Message -----
> > > > >> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
> > > > >> > To: gluster-users at gluster.org, gluster-devel at gluster.org
> > > > >> > Sent: Thursday, March 3, 2016 12:10:42 PM
> > > > >> > Subject: [Gluster-users] gluster volume heal info split brain
> > > command
> > > > >> not     showing files in split-brain
> > > > >> >
> > > > >> >
> > > > >> > Hello,
> > > > >> >
> > > > >> > In gluster, we use the command "gluster volume heal c_glusterfs
> > > > >> > info
> > > > >> > split-brain" to find the files that are in split-brain scenario.
> > > > >> > We run heal script (developed by Windriver prime team) on the
> > > > >> > files
> > > > >> reported
> > > > >> > by above command to resolve split-brain issue.
> > > > >> >
> > > > >> > But we observed that the above command is not showing all files
> > > > >> > that
> > > > >> are in
> > > > >> > split-brain,
> > > > >> > even though split brain scenario actually exists on the node.
> > > > >> >
> > > > >> > Now a days this issue is seen more often and IO errors are
> > > > >> > reported
> > > when
> > > > >> > tried to access these files under split-brain.
> > > > >> >
> > > > >> > Can you please check why this gluster command is not showing files
> > > under
> > > > >> > split-brain?
> > > > >> > We can provide you required logs and support to resolve this
> > > > >> > issue.
> > > > >> Hi,
> > > > >>
> > > > >> Could you paste the output of getfattr -m. -de hex
> > > > >> <path-to-files-in-split-brain> from all the bricks that the files
> > > > >> lie
> > > in?
> > > > >>
> > > > >> >
> > > > >> > Please reply on this because I am not getting any reply from the
> > > > >> community.
> > > > >> >
> > > > >> > --
> > > > >> >
> > > > >> > Regards
> > > > >> > Abhishek Paliwal
> > > > >> >
> > > > >> > _______________________________________________
> > > > >> > Gluster-users mailing list
> > > > >> > Gluster-users at gluster.org
> > > > >> > http://www.gluster.org/mailman/listinfo/gluster-users
> > > > >>
> > > > >> --
> > > > >> Thanks,
> > > > >> Anuradha.
> > > > >>
> > > > >
> > > >
> > >
> > > --
> > > Thanks,
> > > Anuradha.
> > >
> > 
> > 
> > 
> > --
> > 
> > 
> > 
> > 
> > Regards
> > Abhishek Paliwal
> > 
> 
> --
> Thanks,
> Anuradha.
> 

-- 
Thanks,
Anuradha.

More information about the Gluster-users mailing list