[Gluster-users] Determine og force healing occurring

Mon Jul 18 16:52:48 UTC 2016

On Sat, Jul 16, 2016 at 9:53 PM, Jesper Led Lauridsen TS Infra server <
jly at dr.dk> wrote:

> On 07/16/2016 04:10 AM, Pranith Kumar Karampuri wrote:
>
>
>
> On Fri, Jul 15, 2016 at 5:20 PM, Jesper Led Lauridsen TS Infra server <
> <JLY at dr.dk>JLY at dr.dk> wrote:
>
>> Hi,
>>
>> How do I determine in which log etc. that a healing is in progress or
>> startet and how do I if not startet force it.
>>
>> Additional info, is that I have some problem with a volume if I execute
>> 'gluster volume heal <volume> info' the command just hangs but if I
>> execute  'gluster volume heal <volume> info split-brain' it return that no
>> file are in split-brain. Yet there is and I have successfully recovered
>> another one.
>>
>
> If the command hangs there is a chance that operations on the file may
> have lead to stale locks. Could you give the output of statedump?
> You can follow
> https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/ to
> generat the files.
>
>
> Thanks for you response. You are right there was a stale lock. But I am
> sorry I booted all my cluster nodes, so I guess (without knowing) that
> there is no reason to give you the output of a statedump?
>
> What I can confirm and give of information is:
>   * All the servers failed to reboot so I had to push the button. They all
> failed with the message
>      "Unmounting pipe file system: Cannot create link /etc/mtab~
>      Perhaps there is a stale lock file?"
>   * After 2 nodes had rebooted the command executed without any problem
> and reported a couple off split-brain (Both Directory and Files)
>   * strace the command showed that it was just looping, so basically the
> command didn't hanging. It just couldn't finish.
>   * I am using "glusterfs-3.6.2-1.el6.x86_64". But hoping to upgrade to
> 3.6.9 this weekend.
>   * The file I refereed to here. Now has the same output on both replicas
> when getting getfattr information. The
> grusted.afr.glu_rhevtst_dr2_data_01-client-[0,1] and trusted.afr.dirty are
> now all zero
>

If you are anyway looking to upgrade, why not upgrade to 3.7.13, which is
the latest stable version?

>
>
>
>
>> I just have problem with this one. I can determine if there is a healing
>> process running or not
>>
>> I have change 'trusted.afr.glu_rhevtst_dr2_data_01-client-1' to
>> 0x000000000000000000000000 on the file located on glustertst03 and executed
>> a 'ls -lrt' on the file on the gluster-mount.
>>
>> [root at glustertst04 ]# getfattr -d -m . -e hex
>> /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>> getfattr: Removing leading '/' from absolute path names
>> # file:
>> bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>>
>> security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
>> trusted.afr.dirty=0x000000000000000000000000
>> trusted.afr.glu_rhevtst_dr2_data_01-client-0=0x00004c700000000000000000
>> trusted.afr.glu_rhevtst_dr2_data_01-client-1=0x000000000000000000000000
>> trusted.gfid=0x7575f870875b4c899fd81ef16be3b1a1
>>
>> trusted.glusterfs.quota.70145d52-bb80-42ce-b437-64be6ee4a7d4.contri=0x00000001606dc000
>> trusted.pgfid.70145d52-bb80-42ce-b437-64be6ee4a7d4=0x00000001
>>
>>  [root at glustertst03 ]# getfattr -d -m . -e hex
>> /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>> getfattr: Removing leading '/' from absolute path names
>> # file:
>> bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>>
>> security.selinux=0x73797374656d5f753a6f626a6563745f723a66696c655f743a733000
>> trusted.afr.dirty=0x000000270000000000000000
>> trusted.afr.glu_rhevtst_dr2_data_01-client-0=0x000000000000000000000000
>> trusted.afr.glu_rhevtst_dr2_data_01-client-1=0x000000000000000000000000
>> trusted.gfid=0x7575f870875b4c899fd81ef16be3b1a1
>>
>> trusted.glusterfs.quota.70145d52-bb80-42ce-b437-64be6ee4a7d4.contri=0x0000000160662000
>> trusted.pgfid.70145d52-bb80-42ce-b437-64be6ee4a7d4=0x00000001
>>
>> [root at glustertst04 ]# stat
>> /var/run/gluster/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>>   File:
>> `/var/run/gluster/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab'
>>   Size: 21474836480     Blocks: 11548384   IO Block: 131072 regular file
>> Device: 31h/49d Inode: 11517990069246079393  Links: 1
>> Access: (0660/-rw-rw----)  Uid: (   36/    vdsm)   Gid: (   36/     kvm)
>> Access: 2016-07-15 13:33:47.860224289 +0200
>> Modify: 2016-07-15 13:34:44.396125458 +0200
>> Change: 2016-07-15 13:34:44.397125492 +0200
>>
>> [root at glustertst03 ]# stat
>> /bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab
>>   File:
>> `/bricks/brick1/glu_rhevtst_dr2_data_01/6bdc67d1-4ae5-47e3-86c3-ef0916996862/images/7669ca25-028e-40a5-9dc8-06c716101709/a1ae3612-bb89-45d8-8041-134c34592eab'
>>   Size: 21474836480     Blocks: 11547408   IO Block: 4096   regular file
>> Device: fd02h/64770d    Inode: 159515      Links: 2
>> Access: (0660/-rw-rw----)  Uid: (   36/    vdsm)   Gid: (   36/     kvm)
>> Access: 2016-07-13 08:33:00.000561984 +0200
>> Modify: 2016-07-13 08:32:59.969561154 +0200
>> Change: 2016-07-15 12:52:28.414192052 +0200
>>
>> Thanks
>> Jesper
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
> --
> Pranith
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>

-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160718/3f85c5c4/attachment.html>