[Gluster-users] Heal pending problem

Mon Oct 28 08:53:03 UTC 2019

Hi, I have big problem after network outage I think. I have a volume
which is
corrupted I think. Here are details:

[root at hv-01 ~]# gluster --version
glusterfs 5.3

1. volume info:

[root at hv-01 ~]# gluster volume info engine

Volume Name: engine
Type: Replicate
Volume ID: ca7c567a-1512-4c6c-964b-e9a2ca45b872
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: hv-01.cencert.local:/gluster_bricks/engine/engine
Brick2: hv-02.cencert.local:/gluster_bricks/engine/engine
Brick3: arbiter.cencert.local:/gluster_bricks/engine/engine (arbiter)
Options Reconfigured:
cluster.eager-lock: enable
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
user.cifs: off
network.ping-timeout: 30
network.remote-dio: off
performance.strict-o-direct: on
performance.low-prio-threads: 32
features.shard: on
storage.owner-gid: 36
storage.owner-uid: 36
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

2. Volume heal info and summary

[root at hv-01 ~]# gluster volume heal engine info
Brick hv-01.cencert.local:/gluster_bricks/engine/engine
/7396183b-825b-4670-8646-5eb8d69f3480/images/02d6cf5e-179c-4730-8191-
a1ee7b5a7737/30b0a24e-01cb-4d53-9cf1-422f8f217e33 
Status: Connected
Number of entries: 1

Brick hv-02.cencert.local:/gluster_bricks/engine/engine
/7396183b-825b-4670-8646-5eb8d69f3480/images/02d6cf5e-179c-4730-8191-
a1ee7b5a7737/30b0a24e-01cb-4d53-9cf1-422f8f217e33 
Status: Connected
Number of entries: 1

Brick arbiter.cencert.local:/gluster_bricks/engine/engine
/7396183b-825b-4670-8646-5eb8d69f3480/images/02d6cf5e-179c-4730-8191-
a1ee7b5a7737/30b0a24e-01cb-4d53-9cf1-422f8f217e33 
Status: Connected
Number of entries: 1

[root at hv-01 ~]# gluster volume heal engine info summary
Brick hv-01.cencert.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 1
Number of entries in heal pending: 1
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick hv-02.cencert.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 2
Number of entries in heal pending: 2
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick arbiter.cencert.local:/gluster_bricks/engine/engine
Status: Connected
Total Number of entries: 2
Number of entries in heal pending: 2
Number of entries in split-brain: 0
Number of entries possibly healing: 0

3. Details about corrupted file

[root at hv-01 ~]# stat /gluster_bricks/engine/engine/7396183b-825b-4670-
8646-5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-
01cb-4d53-9cf1-422f8f217e33
  Plik: „/gluster_bricks/engine/engine/7396183b-825b-4670-8646-
5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb-
4d53-9cf1-422f8f217e33”
  rozmiar: 67108864     bloków: 122968     bloki I/O: 4096   zwykły
plik
Urządzenie: fd0fh/64783d        inody: 134349053   dowiązań: 2
Dostęp: (0660/-rw-rw----)  Uid: (   36/    vdsm)   Gid:
(   36/     kvm)
Kontekst: system_u:object_r:unlabeled_t:s0
Dostęp:      2019-02-17 15:00:33.130284553 +0100
Modyfikacja: 2019-02-20 13:15:08.090856642 +0100
Zmiana:      2019-10-25 11:13:15.174957824 +0200
Utworzenie:  -

[root at hv-02 ~]# stat /gluster_bricks/engine/engine/7396183b-825b-4670-
8646-5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-
01cb-4d53-9cf1-422f8f217e33
  Plik: „/gluster_bricks/engine/engine/7396183b-825b-4670-8646-
5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb-
4d53-9cf1-422f8f217e33”
  rozmiar: 67108864     bloków: 109320     bloki I/O: 4096   zwykły
plik
Urządzenie: fd14h/64788d        inody: 201326687   dowiązań: 2
Dostęp: (0660/-rw-rw----)  Uid: (   36/    vdsm)   Gid:
(   36/     kvm)
Kontekst: system_u:object_r:unlabeled_t:s0
Dostęp:      2019-02-17 15:00:33.130284553 +0100
Modyfikacja: 2019-02-20 13:15:08.090856642 +0100
Zmiana:      2019-10-25 11:13:06.424507311 +0200
Utworzenie:  -

[root at hv-01 ~]# getfattr -d -e hex -m .
/gluster_bricks/engine/engine/7396183b-825b-4670-8646-
5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb-
4d53-9cf1-422f8f217e33
getfattr: Usunięcie wiodącego '/' ze ścieżek bezwzględnych
# file: gluster_bricks/engine/engine/7396183b-825b-4670-8646-
5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb-
4d53-9cf1-422f8f217e33
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65
645f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.engine-client-1=0x000000010000000000000000
trusted.afr.engine-client-2=0x000000000000000000000000
trusted.gfid=0xf5af72751c1d498a97b7cee251b7f5a7
trusted.gfid2path.448e0d5596fdc4d9=0x38623630363364642d306562322d346362
392d383038342d3435653132656362353336652f33306230613234652d303163622d346
435332d396366312d343232663866323137653333
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-
size=0x000000200000000000000000000000000000000002bf8cdb0000000000000000

[root at hv-02 ~]# getfattr -d -e hex -m .
/gluster_bricks/engine/engine/7396183b-825b-4670-8646-
5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb-
4d53-9cf1-422f8f217e33
getfattr: Usunięcie wiodącego '/' ze ścieżek bezwzględnych
# file: gluster_bricks/engine/engine/7396183b-825b-4670-8646-
5eb8d69f3480/images/02d6cf5e-179c-4730-8191-a1ee7b5a7737/30b0a24e-01cb-
4d53-9cf1-422f8f217e33
security.selinux=0x73797374656d5f753a6f626a6563745f723a756e6c6162656c65
645f743a733000
trusted.afr.dirty=0x000000010000000000000000
trusted.afr.engine-client-0=0x000000010000000000000000
trusted.afr.engine-client-1=0x000000000000000000000000
trusted.gfid=0xf5af72751c1d498a97b7cee251b7f5a7
trusted.gfid2path.448e0d5596fdc4d9=0x38623630363364642d306562322d346362
392d383038342d3435653132656362353336652f33306230613234652d303163622d346
435332d396366312d343232663866323137653333
trusted.glusterfs.shard.block-size=0x0000000004000000
trusted.glusterfs.shard.file-
size=0x000000200000000000000000000000000000000002bf8cdb0000000000000000

I'm also attaching shd log info.
Can it be fixed? I really, really need to repair it.

regards
grzegorz
-------------- next part --------------
A non-text attachment was scrubbed...
Name: glfsheal-engine.log
Type: text/x-log
Size: 27604 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191028/ac13a81c/attachment.bin>