[Gluster-users] self-heal failed

[Liang Ma]

Hi Daniel,

Ok, if gluster can't self-heal from this situation, I hope at least I can
manually restore the volume by using the good brick available. So would you
please tell me how can I "simply rebuild the filesystem and let gluster
attempt to restore it from a *clean* filesystem"?

Many thanks.

Liang


On Wed, Jan 9, 2013 at 9:04 AM, Daniel Taylor <dtaylor at vocalabs.com> wrote:

> It seems to me that what you need to do is replace the failed brick, or
> simply rebuild the filesystem and let gluster attempt to restore it from a
> *clean* filesystem.
>
> I haven't seen anywhere that allows gluster to actually change the
> replication count on a live cluster, which is what you seem to be
> requesting.
>
>
> On 01/09/2013 07:57 AM, Liang Ma wrote:
>
>> Todd,
>>
>> Thanks for your reply. But how can I take this brick offline? Since the
>> gluster volume has replicate count 2, it won't allow me to remove one
>> brick. Is there a command which can take one replicate brick offline?
>>
>> Many thanks.
>>
>> Liang
>>
>>
>> On Tue, Jan 8, 2013 at 3:02 PM, Todd Pfaff <pfaff at rhpcs.mcmaster.ca<mailto:
>> pfaff at rhpcs.mcmaster.**ca <pfaff at rhpcs.mcmaster.ca>>> wrote:
>>
>>     Liang,
>>
>>     I don't claim to know the answer to your question, and my
>>     knowledge of zfs
>>     is minimal at best so I may be way off base here, but it seems to
>>     me that
>>     your attempted random corruption with this command:
>>
>>
>>       dd if=/dev/urandom of=/dev/sda6 bs=1024 count=20480
>>
>>     is likely going to corrupt the underlying zfs filesystem metadata, not
>>     just file data, and I wouldn't expect gluster to be able to fixed a
>>     brick's corrupted filesystem.  Perhaps you now have to take the brick
>>     offline, fix any zfs filesystem errors if possible, bring the
>>     brick back
>>     online and see what then happens with self-heal.
>>
>>     --
>>     Todd Pfaff <pfaff at mcmaster.ca <mailto:pfaff at mcmaster.ca>>
>>
>>     http://www.rhpcs.mcmaster.ca/
>>
>>
>>     On Tue, 8 Jan 2013, Liang Ma wrote:
>>
>>         Hi There,
>>
>>         I'd like to test and understand the self heal feature of
>>         glusterfs. This is
>>         what I did with 3.3.1-ubuntu1~precise4 on Ubuntu 12.04.1 LTS.
>>
>>         gluster volume create gtest replica 2 gluster3:/zfs-test
>>         gluster4:/zfs-test
>>         where zfs-test is a zfs pool on partition /dev/sda6 in both nodes.
>>
>>         To simulate a random corruption on node gluster3
>>
>>         dd if=/dev/urandom of=/dev/sda6 bs=1024 count=20480
>>
>>         Now zfs detected the corrupted files
>>
>>           pool: zfs-test
>>          state: ONLINE
>>         status: One or more devices has experienced an error resulting
>>         in data
>>                 corruption.  Applications may be affected.
>>         action: Restore the file in question if possible.  Otherwise
>>         restore the
>>                 entire pool from backup.
>>            see: http://zfsonlinux.org/msg/ZFS-**8000-8A<http://zfsonlinux.org/msg/ZFS-8000-8A>
>>          scan: none requested
>>         config:
>>
>>                 NAME        STATE     READ WRITE CKSUM
>>                 zfs-test   ONLINE       0     0 2.29K
>>                   sda6     ONLINE       0     0 4.59K
>>
>>         errors: Permanent errors have been detected in the following
>>         files:
>>
>>                 /zfs-test/<xattrdir>/trusted.**gfid
>>                        /zfs-test/.glusterfs/b0/1e/**
>> b01ec17c-14cc-4999-938b-**b4a71e358b46
>>                        /zfs-test/.glusterfs/b0/1e/**
>> b01ec17c-14cc-4999-938b-**b4a71e358b46/<xat
>>         trdir>/trusted.gfid
>>                        /zfs-test/.glusterfs/dd/8c/**
>> dd8c6797-18c3-4f3b-b1ca-**86def2b578c5/<xat
>>         trdir>/trusted.gfid
>>
>>         Now the gluster log file shows the self heal can't fix the
>>         corruption
>>         [2013-01-08 12:46:03.371214] W
>>         [afr-common.c:1196:afr_detect_**self_heal_by_iatt]
>>         2-gtest-replicate-0:
>>         /K.iso: gfid different on subvolume
>>         [2013-01-08 12:46:03.373539] E
>>         [afr-self-heal-common.c:1419:**afr_sh_common_lookup_cbk]
>>         2-gtest-replicate-0:
>>         Missing Gfids for /K.iso
>>         [2013-01-08 12:46:03.385701] E
>>         [afr-self-heal-common.c:2160:**afr_self_heal_completion_cbk]
>>         2-gtest-replicate-0: background  gfid self-heal failed on /K.iso
>>         [2013-01-08 12:46:03.385760] W [fuse-bridge.c:292:fuse_entry_**
>> cbk]
>>         0-glusterfs-fuse: 11901: LOOKUP() /K.iso => -1 (No data available)
>>
>>         where K.iso is one of the sample files affected by the dd command.
>>
>>         So could anyone tell me what is the best way to repair the
>>         simulated
>>         corruption?
>>
>>         Thank you.
>>
>>         Liang
>>
>>
>>
>>
>>
>> ______________________________**_________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.**org/mailman/listinfo/gluster-**users<http://supercolony.gluster.org/mailman/listinfo/gluster-users>
>>
>
> --
> Daniel Taylor             VP Operations       Vocal Laboratories, Inc
> dtaylor at vocalabs.com                                     612-235-5711
>
> ______________________________**_________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.**org/mailman/listinfo/gluster-**users<http://supercolony.gluster.org/mailman/listinfo/gluster-users>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130109/178370cf/attachment.html>