[Gluster-users] self-heal not working
Ben Turner
bturner at redhat.com
Sun Aug 27 20:27:38 UTC 2017
----- Original Message -----
> From: "mabi" <mabi at protonmail.ch>
> To: "Ravishankar N" <ravishankar at redhat.com>
> Cc: "Ben Turner" <bturner at redhat.com>, "Gluster Users" <gluster-users at gluster.org>
> Sent: Sunday, August 27, 2017 3:15:33 PM
> Subject: Re: [Gluster-users] self-heal not working
>
> Thanks Ravi for your analysis. So as far as I understand nothing to worry
> about but my question now would be: how do I get rid of this file from the
> heal info?
Correct me if I am wrong but clearing this is just a matter of resetting the afr.dirty xattr? @Ravi - Is this correct?
-b
>
> > -------- Original Message --------
> > Subject: Re: [Gluster-users] self-heal not working
> > Local Time: August 27, 2017 3:45 PM
> > UTC Time: August 27, 2017 1:45 PM
> > From: ravishankar at redhat.com
> > To: mabi <mabi at protonmail.ch>
> > Ben Turner <bturner at redhat.com>, Gluster Users <gluster-users at gluster.org>
> >
> > Yes, the shds did pick up the file for healing (I saw messages like " got
> > entry: 1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea") but no error afterwards.
> >
> > Anyway I reproduced it by manually setting the afr.dirty bit for a zero
> > byte file on all 3 bricks. Since there are no afr pending xattrs
> > indicating good/bad copies and all files are zero bytes, the data
> > self-heal algorithm just picks the file with the latest ctime as source.
> > In your case that was the arbiter brick. In the code, there is a check to
> > prevent data heals if arbiter is the source. So heal was not happening and
> > the entries were not removed from heal-info output.
> >
> > Perhaps we should add a check in the code to just remove the entries from
> > heal-info if size is zero bytes in all bricks.
> >
> > -Ravi
> >
> > On 08/25/2017 06:33 PM, mabi wrote:
> >
> >> Hi Ravi,
> >>
> >> Did you get a chance to have a look at the log files I have attached in my
> >> last mail?
> >>
> >> Best,
> >> Mabi
> >>
> >>> -------- Original Message --------
> >>> Subject: Re: [Gluster-users] self-heal not working
> >>> Local Time: August 24, 2017 12:08 PM
> >>> UTC Time: August 24, 2017 10:08 AM
> >>> From: mabi at protonmail.ch
> >>> To: Ravishankar N
> >>> [<ravishankar at redhat.com>](mailto:ravishankar at redhat.com)
> >>> Ben Turner [<bturner at redhat.com>](mailto:bturner at redhat.com), Gluster
> >>> Users [<gluster-users at gluster.org>](mailto:gluster-users at gluster.org)
> >>>
> >>> Thanks for confirming the command. I have now enabled DEBUG
> >>> client-log-level, run a heal and then attached the glustershd log files
> >>> of all 3 nodes in this mail.
> >>>
> >>> The volume concerned is called myvol-pro, the other 3 volumes have no
> >>> problem so far.
> >>>
> >>> Also note that in the mean time it looks like the file has been deleted
> >>> by the user and as such the heal info command does not show the file
> >>> name anymore but just is GFID which is:
> >>>
> >>> gfid:1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea
> >>>
> >>> Hope that helps for debugging this issue.
> >>>
> >>>> -------- Original Message --------
> >>>> Subject: Re: [Gluster-users] self-heal not working
> >>>> Local Time: August 24, 2017 5:58 AM
> >>>> UTC Time: August 24, 2017 3:58 AM
> >>>> From: ravishankar at redhat.com
> >>>> To: mabi [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)
> >>>> Ben Turner [<bturner at redhat.com>](mailto:bturner at redhat.com), Gluster
> >>>> Users [<gluster-users at gluster.org>](mailto:gluster-users at gluster.org)
> >>>>
> >>>> Unlikely. In your case only the afr.dirty is set, not the
> >>>> afr.volname-client-xx xattr.
> >>>>
> >>>> `gluster volume set myvolume diagnostics.client-log-level DEBUG` is
> >>>> right.
> >>>>
> >>>> On 08/23/2017 10:31 PM, mabi wrote:
> >>>>
> >>>>> I just saw the following bug which was fixed in 3.8.15:
> >>>>>
> >>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1471613
> >>>>>
> >>>>> Is it possible that the problem I described in this post is related to
> >>>>> that bug?
> >>>>>
> >>>>>> -------- Original Message --------
> >>>>>> Subject: Re: [Gluster-users] self-heal not working
> >>>>>> Local Time: August 22, 2017 11:51 AM
> >>>>>> UTC Time: August 22, 2017 9:51 AM
> >>>>>> From: ravishankar at redhat.com
> >>>>>> To: mabi [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)
> >>>>>> Ben Turner [<bturner at redhat.com>](mailto:bturner at redhat.com), Gluster
> >>>>>> Users [<gluster-users at gluster.org>](mailto:gluster-users at gluster.org)
> >>>>>>
> >>>>>> On 08/22/2017 02:30 PM, mabi wrote:
> >>>>>>
> >>>>>>> Thanks for the additional hints, I have the following 2 questions
> >>>>>>> first:
> >>>>>>>
> >>>>>>> - In order to launch the index heal is the following command correct:
> >>>>>>> gluster volume heal myvolume
> >>>>>>
> >>>>>> Yes
> >>>>>>
> >>>>>>> - If I run a "volume start force" will it have any short disruptions
> >>>>>>> on my clients which mount the volume through FUSE? If yes, how long?
> >>>>>>> This is a production system that's why I am asking.
> >>>>>>
> >>>>>> No. You can actually create a test volume on your personal linux box
> >>>>>> to try these kinds of things without needing multiple machines. This
> >>>>>> is how we develop and test our patches :)
> >>>>>> 'gluster volume create testvol replica 3 /home/mabi/bricks/brick{1..3}
> >>>>>> force` and so on.
> >>>>>>
> >>>>>> HTH,
> >>>>>> Ravi
> >>>>>>
> >>>>>>>> -------- Original Message --------
> >>>>>>>> Subject: Re: [Gluster-users] self-heal not working
> >>>>>>>> Local Time: August 22, 2017 6:26 AM
> >>>>>>>> UTC Time: August 22, 2017 4:26 AM
> >>>>>>>> From: ravishankar at redhat.com
> >>>>>>>> To: mabi [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch), Ben
> >>>>>>>> Turner [<bturner at redhat.com>](mailto:bturner at redhat.com)
> >>>>>>>> Gluster Users
> >>>>>>>> [<gluster-users at gluster.org>](mailto:gluster-users at gluster.org)
> >>>>>>>>
> >>>>>>>> Explore the following:
> >>>>>>>>
> >>>>>>>> - Launch index heal and look at the glustershd logs of all bricks
> >>>>>>>> for possible errors
> >>>>>>>>
> >>>>>>>> - See if the glustershd in each node is connected to all bricks.
> >>>>>>>>
> >>>>>>>> - If not try to restart shd by `volume start force`
> >>>>>>>>
> >>>>>>>> - Launch index heal again and try.
> >>>>>>>>
> >>>>>>>> - Try debugging the shd log by setting client-log-level to DEBUG
> >>>>>>>> temporarily.
> >>>>>>>>
> >>>>>>>> On 08/22/2017 03:19 AM, mabi wrote:
> >>>>>>>>
> >>>>>>>>> Sure, it doesn't look like a split brain based on the output:
> >>>>>>>>>
> >>>>>>>>> Brick node1.domain.tld:/data/myvolume/brick
> >>>>>>>>> Status: Connected
> >>>>>>>>> Number of entries in split-brain: 0
> >>>>>>>>>
> >>>>>>>>> Brick node2.domain.tld:/data/myvolume/brick
> >>>>>>>>> Status: Connected
> >>>>>>>>> Number of entries in split-brain: 0
> >>>>>>>>>
> >>>>>>>>> Brick node3.domain.tld:/srv/glusterfs/myvolume/brick
> >>>>>>>>> Status: Connected
> >>>>>>>>> Number of entries in split-brain: 0
> >>>>>>>>>
> >>>>>>>>>> -------- Original Message --------
> >>>>>>>>>> Subject: Re: [Gluster-users] self-heal not working
> >>>>>>>>>> Local Time: August 21, 2017 11:35 PM
> >>>>>>>>>> UTC Time: August 21, 2017 9:35 PM
> >>>>>>>>>> From: bturner at redhat.com
> >>>>>>>>>> To: mabi [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)
> >>>>>>>>>> Gluster Users
> >>>>>>>>>> [<gluster-users at gluster.org>](mailto:gluster-users at gluster.org)
> >>>>>>>>>>
> >>>>>>>>>> Can you also provide:
> >>>>>>>>>>
> >>>>>>>>>> gluster v heal <my vol> info split-brain
> >>>>>>>>>>
> >>>>>>>>>> If it is split brain just delete the incorrect file from the brick
> >>>>>>>>>> and run heal again. I haven"t tried this with arbiter but I
> >>>>>>>>>> assume the process is the same.
> >>>>>>>>>>
> >>>>>>>>>> -b
> >>>>>>>>>>
> >>>>>>>>>> ----- Original Message -----
> >>>>>>>>>>> From: "mabi" [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)
> >>>>>>>>>>> To: "Ben Turner"
> >>>>>>>>>>> [<bturner at redhat.com>](mailto:bturner at redhat.com)
> >>>>>>>>>>> Cc: "Gluster Users"
> >>>>>>>>>>> [<gluster-users at gluster.org>](mailto:gluster-users at gluster.org)
> >>>>>>>>>>> Sent: Monday, August 21, 2017 4:55:59 PM
> >>>>>>>>>>> Subject: Re: [Gluster-users] self-heal not working
> >>>>>>>>>>>
> >>>>>>>>>>> Hi Ben,
> >>>>>>>>>>>
> >>>>>>>>>>> So it is really a 0 kBytes file everywhere (all nodes including
> >>>>>>>>>>> the arbiter
> >>>>>>>>>>> and from the client).
> >>>>>>>>>>> Here below you will find the output you requested. Hopefully that
> >>>>>>>>>>> will help
> >>>>>>>>>>> to find out why this specific file is not healing... Let me know
> >>>>>>>>>>> if you need
> >>>>>>>>>>> any more information. Btw node3 is my arbiter node.
> >>>>>>>>>>>
> >>>>>>>>>>> NODE1:
> >>>>>>>>>>>
> >>>>>>>>>>> STAT:
> >>>>>>>>>>> File:
> >>>>>>>>>>> ‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’
> >>>>>>>>>>> Size: 0 Blocks: 38 IO Block: 131072 regular empty file
> >>>>>>>>>>> Device: 24h/36d Inode: 10033884 Links: 2
> >>>>>>>>>>> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
> >>>>>>>>>>> Access: 2017-08-14 17:04:55.530681000 +0200
> >>>>>>>>>>> Modify: 2017-08-14 17:11:46.407404779 +0200
> >>>>>>>>>>> Change: 2017-08-14 17:11:46.407404779 +0200
> >>>>>>>>>>> Birth: -
> >>>>>>>>>>>
> >>>>>>>>>>> GETFATTR:
> >>>>>>>>>>> trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
> >>>>>>>>>>> trusted.bit-rot.version=0sAgAAAAAAAABZhuknAAlJAg==
> >>>>>>>>>>> trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
> >>>>>>>>>>> trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOyo=
> >>>>>>>>>>>
> >>>>>>>>>>> NODE2:
> >>>>>>>>>>>
> >>>>>>>>>>> STAT:
> >>>>>>>>>>> File:
> >>>>>>>>>>> ‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’
> >>>>>>>>>>> Size: 0 Blocks: 38 IO Block: 131072 regular empty file
> >>>>>>>>>>> Device: 26h/38d Inode: 10031330 Links: 2
> >>>>>>>>>>> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
> >>>>>>>>>>> Access: 2017-08-14 17:04:55.530681000 +0200
> >>>>>>>>>>> Modify: 2017-08-14 17:11:46.403704181 +0200
> >>>>>>>>>>> Change: 2017-08-14 17:11:46.403704181 +0200
> >>>>>>>>>>> Birth: -
> >>>>>>>>>>>
> >>>>>>>>>>> GETFATTR:
> >>>>>>>>>>> trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
> >>>>>>>>>>> trusted.bit-rot.version=0sAgAAAAAAAABZhu6wAA8Hpw==
> >>>>>>>>>>> trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
> >>>>>>>>>>> trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOVE=
> >>>>>>>>>>>
> >>>>>>>>>>> NODE3:
> >>>>>>>>>>> STAT:
> >>>>>>>>>>> File:
> >>>>>>>>>>> /srv/glusterfs/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
> >>>>>>>>>>> Size: 0 Blocks: 0 IO Block: 4096 regular empty file
> >>>>>>>>>>> Device: ca11h/51729d Inode: 405208959 Links: 2
> >>>>>>>>>>> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
> >>>>>>>>>>> Access: 2017-08-14 17:04:55.530681000 +0200
> >>>>>>>>>>> Modify: 2017-08-14 17:04:55.530681000 +0200
> >>>>>>>>>>> Change: 2017-08-14 17:11:46.604380051 +0200
> >>>>>>>>>>> Birth: -
> >>>>>>>>>>>
> >>>>>>>>>>> GETFATTR:
> >>>>>>>>>>> trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
> >>>>>>>>>>> trusted.bit-rot.version=0sAgAAAAAAAABZe6ejAAKPAg==
> >>>>>>>>>>> trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
> >>>>>>>>>>> trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOc4=
> >>>>>>>>>>>
> >>>>>>>>>>> CLIENT GLUSTER MOUNT:
> >>>>>>>>>>> STAT:
> >>>>>>>>>>> File:
> >>>>>>>>>>> "/mnt/myvolume/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png"
> >>>>>>>>>>> Size: 0 Blocks: 0 IO Block: 131072 regular empty file
> >>>>>>>>>>> Device: 1eh/30d Inode: 11897049013408443114 Links: 1
> >>>>>>>>>>> Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
> >>>>>>>>>>> Access: 2017-08-14 17:04:55.530681000 +0200
> >>>>>>>>>>> Modify: 2017-08-14 17:11:46.407404779 +0200
> >>>>>>>>>>> Change: 2017-08-14 17:11:46.407404779 +0200
> >>>>>>>>>>> Birth: -
> >>>>>>>>>>>
> >>>>>>>>>>> > -------- Original Message --------
> >>>>>>>>>>> > Subject: Re: [Gluster-users] self-heal not working
> >>>>>>>>>>> > Local Time: August 21, 2017 9:34 PM
> >>>>>>>>>>> > UTC Time: August 21, 2017 7:34 PM
> >>>>>>>>>>> > From: bturner at redhat.com
> >>>>>>>>>>> > To: mabi [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)
> >>>>>>>>>>> > Gluster Users
> >>>>>>>>>>> > [<gluster-users at gluster.org>](mailto:gluster-users at gluster.org)
> >>>>>>>>>>> >
> >>>>>>>>>>> > ----- Original Message -----
> >>>>>>>>>>> >> From: "mabi" [<mabi at protonmail.ch>](mailto:mabi at protonmail.ch)
> >>>>>>>>>>> >> To: "Gluster Users"
> >>>>>>>>>>> >> [<gluster-users at gluster.org>](mailto:gluster-users at gluster.org)
> >>>>>>>>>>> >> Sent: Monday, August 21, 2017 9:28:24 AM
> >>>>>>>>>>> >> Subject: [Gluster-users] self-heal not working
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> Hi,
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> I have a replicat 2 with arbiter GlusterFS 3.8.11 cluster and
> >>>>>>>>>>> >> there is
> >>>>>>>>>>> >> currently one file listed to be healed as you can see below
> >>>>>>>>>>> >> but never gets
> >>>>>>>>>>> >> healed by the self-heal daemon:
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> Brick node1.domain.tld:/data/myvolume/brick
> >>>>>>>>>>> >> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
> >>>>>>>>>>> >> Status: Connected
> >>>>>>>>>>> >> Number of entries: 1
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> Brick node2.domain.tld:/data/myvolume/brick
> >>>>>>>>>>> >> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
> >>>>>>>>>>> >> Status: Connected
> >>>>>>>>>>> >> Number of entries: 1
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> Brick node3.domain.tld:/srv/glusterfs/myvolume/brick
> >>>>>>>>>>> >> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
> >>>>>>>>>>> >> Status: Connected
> >>>>>>>>>>> >> Number of entries: 1
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> As once recommended on this mailing list I have mounted that
> >>>>>>>>>>> >> glusterfs
> >>>>>>>>>>> >> volume
> >>>>>>>>>>> >> temporarily through fuse/glusterfs and ran a "stat" on that
> >>>>>>>>>>> >> file which is
> >>>>>>>>>>> >> listed above but nothing happened.
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> The file itself is available on all 3 nodes/bricks but on the
> >>>>>>>>>>> >> last node it
> >>>>>>>>>>> >> has a different date. By the way this file is 0 kBytes big. Is
> >>>>>>>>>>> >> that maybe
> >>>>>>>>>>> >> the reason why the self-heal does not work?
> >>>>>>>>>>> >
> >>>>>>>>>>> > Is the file actually 0 bytes or is it just 0 bytes on the
> >>>>>>>>>>> > arbiter(0 bytes
> >>>>>>>>>>> > are expected on the arbiter, it just stores metadata)? Can you
> >>>>>>>>>>> > send us the
> >>>>>>>>>>> > output from stat on all 3 nodes:
> >>>>>>>>>>> >
> >>>>>>>>>>> > $ stat <file on back end brick>
> >>>>>>>>>>> > $ getfattr -d -m - <file on back end brick>
> >>>>>>>>>>> > $ stat <file from gluster mount>
> >>>>>>>>>>> >
> >>>>>>>>>>> > Lets see what things look like on the back end, it should tell
> >>>>>>>>>>> > us why
> >>>>>>>>>>> > healing is failing.
> >>>>>>>>>>> >
> >>>>>>>>>>> > -b
> >>>>>>>>>>> >
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> And how can I now make this file to heal?
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> Thanks,
> >>>>>>>>>>> >> Mabi
> >>>>>>>>>>> >>
> >>>>>>>>>>> >>
> >>>>>>>>>>> >>
> >>>>>>>>>>> >>
> >>>>>>>>>>> >> _______________________________________________
> >>>>>>>>>>> >> Gluster-users mailing list
> >>>>>>>>>>> >> Gluster-users at gluster.org
> >>>>>>>>>>> >> http://lists.gluster.org/mailman/listinfo/gluster-users
> >>>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> Gluster-users mailing list
> >>>>>>>>> Gluster-users at gluster.org
> >>>>>>>>>
> >>>>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list