[Gluster-devel] [Gluster-users] gluster volume heal info split brain command not showing files in split-brain

Wed Mar 16 11:09:26 UTC 2016

Hi Anuradha,

I am doing the same which is mentioned in the link you shared and It has
been resolved the issue.

But my question is if it is the split-brain scenario then why the command
"gluster volume heal info split-brain"
 not showing these files in the output even not the parent directory is
present in split-brain.

Please find the requested logs

On Wed, Mar 16, 2016 at 4:20 PM, Anuradha Talur <atalur at redhat.com> wrote:

> Hi Abhishek,
>
> The files that are reporting i/o error have gfid-mismatch. This situation
> is called directory or entry split-brain. You can find steps to resolve
> this kind of split brain here :
> https://gluster.readthedocs.org/en/latest/Troubleshooting/split-brain/ .
>
> Ideally, the parent directories of these files have to be listed in heal
> info split-brain output. Can you please get extended attributes of parent
> directories of the files that show i/o error (Same getfattr command that
> you previously used.) ?
>
> ----- Original Message -----
> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
> > To: "Anuradha Talur" <atalur at redhat.com>
> > Cc: gluster-users at gluster.org, gluster-devel at gluster.org
> > Sent: Thursday, March 10, 2016 11:22:35 AM
> > Subject: Re: [Gluster-users] gluster volume heal info split brain
> command not showing files in split-brain
> >
> > Hi Anuradha,
> >
> > Please find the glusterfs and glusterd logs directory as an attachment.
> >
> > Regards,
> > Abhishek
> >
> >
> >
> > On Wed, Mar 9, 2016 at 5:54 PM, ABHISHEK PALIWAL <
> abhishpaliwal at gmail.com>
> > wrote:
> >
> > > Hi Anuradha,
> > >
> > > Sorry for late reply.
> > >
> > > Please find the requested logs below:
> > >
> > > Remote: 10.32.0.48
> > > Local : 10.32.1.144
> > >
> > > Local:
> > > #gluster volume heal c_glusterfs info split-brain
> > > Brick 10.32.1.144:/opt/lvmdir/c2/brick
> > > Number of entries in split-brain: 0
> > >
> > > Brick 10.32.0.48:/opt/lvmdir/c2/brick
> > > Number of entries in split-brain: 0
> > >
> > > Remote:
> > > #gluster volume heal c_glusterfs info split-brain
> > > Brick 10.32.1.144:/opt/lvmdir/c2/brick
> > > Number of entries in split-brain: 0
> > >
> > > Brick 10.32.0.48:/opt/lvmdir/c2/brick
> > > Number of entries in split-brain: 0
> > >
> > > auto-sync.sh.
> > > Here you can see that i/o error is detected. Below is the required meta
> > > data from both the bricks.
> > >
> > > 1)
> > > stat: cannot stat '/mnt/c//public_html/cello/ior_files/nameroot.ior':
> > > Input/output error
> > > Remote:
> > >
> > > getfattr -d -m . -e hex
> > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > > trusted.afr.dirty=0x000000000000000000000000
> > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f
> > > trusted.gfid=0x771221a7bb3c4f1aade40ce9e38a95ee
> > >
> > > Local:
> > >
> > > getfattr -d -m . -e hex
> > > opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > > # file: opt/lvmdir/c2/brick/public_html/cello/ior_files/nameroot.ior
> > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51
> > > trusted.gfid=0x8ea33f46703c4e2d95c09153c1b858fd
> > >
> > >
> > > 2)
> > > stat: cannot stat '/mnt/c//security/corbasecurity': Input/output error
> > > Remote:
> > >
> > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
> > > # file: opt/lvmdir/c2/brick/security/corbasecurity
> > > trusted.afr.dirty=0x000000000000000000000000
> > > trusted.bit-rot.version=0x000000000000000256ded2f6000ad80f
> > > trusted.gfid=0xd298b7a0c8834f3e99abb39741363013
> > >
> > > Local:
> > >
> > > getfattr -d -m . -e hex opt/lvmdir/c2/brick/security/corbasecurity
> > > # file: opt/lvmdir/c2/brick/security/corbasecurity
> > > trusted.bit-rot.version=0x000000000000000256ded38f000e3a51
> > > trusted.gfid=0x890df0f706184b52803fac3242a2f15b
> > >
> > > I observed that getfattr command output doesn't show all the fields all
> > > the times.
> > >
> > > Here you can check that gluster split-brain command hasn't reported any
> > > split-brains but resulted in IO errors when accessed few files.
> > >
> > > Could you please tell me if "split-brain" command doesn't reported any
> > > entry as output, then is there any way through which we can find out
> that
> > > the files are in split-brain if we are getting the IO error on those
> file.
> > >
> > >
> > > Regards,
> > > Abhishek
> > >
> > >
> > >
> > > On Thu, Mar 3, 2016 at 5:32 PM, Anuradha Talur <atalur at redhat.com>
> wrote:
> > >
> > >>
> > >>
> > >> ----- Original Message -----
> > >> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
> > >> > To: gluster-users at gluster.org, gluster-devel at gluster.org
> > >> > Sent: Thursday, March 3, 2016 12:10:42 PM
> > >> > Subject: [Gluster-users] gluster volume heal info split brain
> command
> > >> not     showing files in split-brain
> > >> >
> > >> >
> > >> > Hello,
> > >> >
> > >> > In gluster, we use the command "gluster volume heal c_glusterfs info
> > >> > split-brain" to find the files that are in split-brain scenario.
> > >> > We run heal script (developed by Windriver prime team) on the files
> > >> reported
> > >> > by above command to resolve split-brain issue.
> > >> >
> > >> > But we observed that the above command is not showing all files that
> > >> are in
> > >> > split-brain,
> > >> > even though split brain scenario actually exists on the node.
> > >> >
> > >> > Now a days this issue is seen more often and IO errors are reported
> when
> > >> > tried to access these files under split-brain.
> > >> >
> > >> > Can you please check why this gluster command is not showing files
> under
> > >> > split-brain?
> > >> > We can provide you required logs and support to resolve this issue.
> > >> Hi,
> > >>
> > >> Could you paste the output of getfattr -m. -de hex
> > >> <path-to-files-in-split-brain> from all the bricks that the files lie
> in?
> > >>
> > >> >
> > >> > Please reply on this because I am not getting any reply from the
> > >> community.
> > >> >
> > >> > --
> > >> >
> > >> > Regards
> > >> > Abhishek Paliwal
> > >> >
> > >> > _______________________________________________
> > >> > Gluster-users mailing list
> > >> > Gluster-users at gluster.org
> > >> > http://www.gluster.org/mailman/listinfo/gluster-users
> > >>
> > >> --
> > >> Thanks,
> > >> Anuradha.
> > >>
> > >
> >
>
> --
> Thanks,
> Anuradha.
>

-- 

Regards
Abhishek Paliwal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160316/1cecd830/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: split-brain.log
Type: text/x-log
Size: 6716 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160316/1cecd830/attachment-0001.bin>