[Gluster-users] self-heal not working

Thu Aug 24 03:58:58 UTC 2017

Unlikely. In your case only the afr.dirty is set, not the 
afr.volname-client-xx xattr.

`gluster volume set myvolume diagnostics.client-log-level DEBUG` is right.

On 08/23/2017 10:31 PM, mabi wrote:
> I just saw the following bug which was fixed in 3.8.15:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1471613
>
> Is it possible that the problem I described in this post is related to 
> that bug?
>
>
>
>> -------- Original Message --------
>> Subject: Re: [Gluster-users] self-heal not working
>> Local Time: August 22, 2017 11:51 AM
>> UTC Time: August 22, 2017 9:51 AM
>> From: ravishankar at redhat.com
>> To: mabi <mabi at protonmail.ch>
>> Ben Turner <bturner at redhat.com>, Gluster Users 
>> <gluster-users at gluster.org>
>>
>>
>>
>>
>> On 08/22/2017 02:30 PM, mabi wrote:
>>> Thanks for the additional hints, I have the following 2 questions first:
>>>
>>> - In order to launch the index heal is the following command correct:
>>> gluster volume heal myvolume
>>>
>> Yes
>>
>>> - If I run a "volume start force" will it have any short disruptions 
>>> on my clients which mount the volume through FUSE? If yes, how long? 
>>> This is a production system that's why I am asking.
>>>
>>>
>> No. You can actually create a test volume on  your personal linux box 
>> to try these kinds of things without needing multiple machines. This 
>> is how we develop and test our patches :)
>> 'gluster volume create testvol replica 3 
>> /home/mabi/bricks/brick{1..3} force` and so on.
>>
>> HTH,
>> Ravi
>>
>>
>>>
>>>> -------- Original Message --------
>>>> Subject: Re: [Gluster-users] self-heal not working
>>>> Local Time: August 22, 2017 6:26 AM
>>>> UTC Time: August 22, 2017 4:26 AM
>>>> From: ravishankar at redhat.com
>>>> To: mabi <mabi at protonmail.ch>, Ben Turner <bturner at redhat.com>
>>>> Gluster Users <gluster-users at gluster.org>
>>>>
>>>>
>>>> Explore the following:
>>>>
>>>> - Launch index heal and look at the glustershd logs of all bricks 
>>>> for possible errors
>>>>
>>>> - See if the glustershd in each node is connected to all bricks.
>>>>
>>>> - If not try to restart shd by `volume start force`
>>>>
>>>> - Launch index heal again and try.
>>>>
>>>> - Try debugging the shd log by setting client-log-level to DEBUG 
>>>> temporarily.
>>>>
>>>>
>>>> On 08/22/2017 03:19 AM, mabi wrote:
>>>>> Sure, it doesn't look like a split brain based on the output:
>>>>>
>>>>> Brick node1.domain.tld:/data/myvolume/brick
>>>>> Status: Connected
>>>>> Number of entries in split-brain: 0
>>>>>
>>>>> Brick node2.domain.tld:/data/myvolume/brick
>>>>> Status: Connected
>>>>> Number of entries in split-brain: 0
>>>>>
>>>>> Brick node3.domain.tld:/srv/glusterfs/myvolume/brick
>>>>> Status: Connected
>>>>> Number of entries in split-brain: 0
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> -------- Original Message --------
>>>>>> Subject: Re: [Gluster-users] self-heal not working
>>>>>> Local Time: August 21, 2017 11:35 PM
>>>>>> UTC Time: August 21, 2017 9:35 PM
>>>>>> From: bturner at redhat.com
>>>>>> To: mabi <mabi at protonmail.ch>
>>>>>> Gluster Users <gluster-users at gluster.org>
>>>>>>
>>>>>> Can you also provide:
>>>>>>
>>>>>> gluster v heal <my vol> info split-brain
>>>>>>
>>>>>> If it is split brain just delete the incorrect file from the 
>>>>>> brick and run heal again. I haven"t tried this with arbiter but I 
>>>>>> assume the process is the same.
>>>>>>
>>>>>> -b
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> > From: "mabi" <mabi at protonmail.ch>
>>>>>> > To: "Ben Turner" <bturner at redhat.com>
>>>>>> > Cc: "Gluster Users" <gluster-users at gluster.org>
>>>>>> > Sent: Monday, August 21, 2017 4:55:59 PM
>>>>>> > Subject: Re: [Gluster-users] self-heal not working
>>>>>> >
>>>>>> > Hi Ben,
>>>>>> >
>>>>>> > So it is really a 0 kBytes file everywhere (all nodes including 
>>>>>> the arbiter
>>>>>> > and from the client).
>>>>>> > Here below you will find the output you requested. Hopefully 
>>>>>> that will help
>>>>>> > to find out why this specific file is not healing... Let me 
>>>>>> know if you need
>>>>>> > any more information. Btw node3 is my arbiter node.
>>>>>> >
>>>>>> > NODE1:
>>>>>> >
>>>>>> > STAT:
>>>>>> > File:
>>>>>> > 
>>>>>> ‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’
>>>>>> > Size: 0 Blocks: 38 IO Block: 131072 regular empty file
>>>>>> > Device: 24h/36d Inode: 10033884 Links: 2
>>>>>> > Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>> > Access: 2017-08-14 17:04:55.530681000 +0200
>>>>>> > Modify: 2017-08-14 17:11:46.407404779 +0200
>>>>>> > Change: 2017-08-14 17:11:46.407404779 +0200
>>>>>> > Birth: -
>>>>>> >
>>>>>> > GETFATTR:
>>>>>> > trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>>>> > trusted.bit-rot.version=0sAgAAAAAAAABZhuknAAlJAg==
>>>>>> > trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
>>>>>> > 
>>>>>> trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOyo=
>>>>>> >
>>>>>> > NODE2:
>>>>>> >
>>>>>> > STAT:
>>>>>> > File:
>>>>>> > 
>>>>>> ‘/data/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png’
>>>>>> > Size: 0 Blocks: 38 IO Block: 131072 regular empty file
>>>>>> > Device: 26h/38d Inode: 10031330 Links: 2
>>>>>> > Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>> > Access: 2017-08-14 17:04:55.530681000 +0200
>>>>>> > Modify: 2017-08-14 17:11:46.403704181 +0200
>>>>>> > Change: 2017-08-14 17:11:46.403704181 +0200
>>>>>> > Birth: -
>>>>>> >
>>>>>> > GETFATTR:
>>>>>> > trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>>>> > trusted.bit-rot.version=0sAgAAAAAAAABZhu6wAA8Hpw==
>>>>>> > trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
>>>>>> > 
>>>>>> trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOVE=
>>>>>> >
>>>>>> > NODE3:
>>>>>> > STAT:
>>>>>> > File:
>>>>>> > 
>>>>>> /srv/glusterfs/myvolume/brick/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>> > Size: 0 Blocks: 0 IO Block: 4096 regular empty file
>>>>>> > Device: ca11h/51729d Inode: 405208959 Links: 2
>>>>>> > Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>> > Access: 2017-08-14 17:04:55.530681000 +0200
>>>>>> > Modify: 2017-08-14 17:04:55.530681000 +0200
>>>>>> > Change: 2017-08-14 17:11:46.604380051 +0200
>>>>>> > Birth: -
>>>>>> >
>>>>>> > GETFATTR:
>>>>>> > trusted.afr.dirty=0sAAAAAQAAAAAAAAAA
>>>>>> > trusted.bit-rot.version=0sAgAAAAAAAABZe6ejAAKPAg==
>>>>>> > trusted.gfid=0sGYXiM9XuTj6lGs8LX58q6g==
>>>>>> > 
>>>>>> trusted.glusterfs.d99af2fa-439b-4a21-bf3a-38f3849f87ec.xtime=0sWZG9sgAGOc4=
>>>>>> >
>>>>>> > CLIENT GLUSTER MOUNT:
>>>>>> > STAT:
>>>>>> > File:
>>>>>> > 
>>>>>> "/mnt/myvolume/data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png"
>>>>>> > Size: 0 Blocks: 0 IO Block: 131072 regular empty file
>>>>>> > Device: 1eh/30d Inode: 11897049013408443114 Links: 1
>>>>>> > Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
>>>>>> > Access: 2017-08-14 17:04:55.530681000 +0200
>>>>>> > Modify: 2017-08-14 17:11:46.407404779 +0200
>>>>>> > Change: 2017-08-14 17:11:46.407404779 +0200
>>>>>> > Birth: -
>>>>>> >
>>>>>> > > -------- Original Message --------
>>>>>> > > Subject: Re: [Gluster-users] self-heal not working
>>>>>> > > Local Time: August 21, 2017 9:34 PM
>>>>>> > > UTC Time: August 21, 2017 7:34 PM
>>>>>> > > From: bturner at redhat.com
>>>>>> > > To: mabi <mabi at protonmail.ch>
>>>>>> > > Gluster Users <gluster-users at gluster.org>
>>>>>> > >
>>>>>> > > ----- Original Message -----
>>>>>> > >> From: "mabi" <mabi at protonmail.ch>
>>>>>> > >> To: "Gluster Users" <gluster-users at gluster.org>
>>>>>> > >> Sent: Monday, August 21, 2017 9:28:24 AM
>>>>>> > >> Subject: [Gluster-users] self-heal not working
>>>>>> > >>
>>>>>> > >> Hi,
>>>>>> > >>
>>>>>> > >> I have a replicat 2 with arbiter GlusterFS 3.8.11 cluster 
>>>>>> and there is
>>>>>> > >> currently one file listed to be healed as you can see below 
>>>>>> but never gets
>>>>>> > >> healed by the self-heal daemon:
>>>>>> > >>
>>>>>> > >> Brick node1.domain.tld:/data/myvolume/brick
>>>>>> > >> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>> > >> Status: Connected
>>>>>> > >> Number of entries: 1
>>>>>> > >>
>>>>>> > >> Brick node2.domain.tld:/data/myvolume/brick
>>>>>> > >> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>> > >> Status: Connected
>>>>>> > >> Number of entries: 1
>>>>>> > >>
>>>>>> > >> Brick node3.domain.tld:/srv/glusterfs/myvolume/brick
>>>>>> > >> /data/appdata_ocpom4nckwru/preview/1344699/64-64-crop.png
>>>>>> > >> Status: Connected
>>>>>> > >> Number of entries: 1
>>>>>> > >>
>>>>>> > >> As once recommended on this mailing list I have mounted that 
>>>>>> glusterfs
>>>>>> > >> volume
>>>>>> > >> temporarily through fuse/glusterfs and ran a "stat" on that 
>>>>>> file which is
>>>>>> > >> listed above but nothing happened.
>>>>>> > >>
>>>>>> > >> The file itself is available on all 3 nodes/bricks but on 
>>>>>> the last node it
>>>>>> > >> has a different date. By the way this file is 0 kBytes big. 
>>>>>> Is that maybe
>>>>>> > >> the reason why the self-heal does not work?
>>>>>> > >
>>>>>> > > Is the file actually 0 bytes or is it just 0 bytes on the 
>>>>>> arbiter(0 bytes
>>>>>> > > are expected on the arbiter, it just stores metadata)? Can 
>>>>>> you send us the
>>>>>> > > output from stat on all 3 nodes:
>>>>>> > >
>>>>>> > > $ stat <file on back end brick>
>>>>>> > > $ getfattr -d -m - <file on back end brick>
>>>>>> > > $ stat <file from gluster mount>
>>>>>> > >
>>>>>> > > Lets see what things look like on the back end, it should 
>>>>>> tell us why
>>>>>> > > healing is failing.
>>>>>> > >
>>>>>> > > -b
>>>>>> > >
>>>>>> > >>
>>>>>> > >> And how can I now make this file to heal?
>>>>>> > >>
>>>>>> > >> Thanks,
>>>>>> > >> Mabi
>>>>>> > >>
>>>>>> > >>
>>>>>> > >>
>>>>>> > >>
>>>>>> > >> _______________________________________________
>>>>>> > >> Gluster-users mailing list
>>>>>> > >> Gluster-users at gluster.org
>>>>>> > >> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170824/36657925/attachment.html>