[Bugs] [Bug 1565623] glusterfs disperse volume input output error
bugzilla at redhat.com
bugzilla at redhat.com
Wed Apr 11 10:52:03 UTC 2018
https://bugzilla.redhat.com/show_bug.cgi?id=1565623
--- Comment #11 from Alexey Shcherbakov <alexey.shcherbakov at kaspersky.com> ---
(In reply to Xavi Hernandez from comment #10)
> All seems to indicate that a heal was happening on nodes glfs-node13.avp.ru
> and glfs-node25.avp.ru at the time of restarting node glfs-node19.avp.ru.
> Unfortunately this coincided with a modification that caused 3 simultaneous
> failures on the file.
>
> We need to manually repair the file or recover it from a backup.
>
> To recover the file manually we have two options:
>
> 1. Guess which of the 3 bad fragments is "less" bad. Probably the best
> candidate would be the fragment on node glfs-node19.avp.ru, but we need to
> check it. It would be interesting to see the modification times of all
> fragments on all bricks. To do so we can execute 'stat
> /data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2'. This will help us to
> decide, but it's not a 100% secure way to determine the best option.
>
> 2. Try to check integrity of fragments. To do this we'll need to develop a
> small tool able to do the check. It will require some time but it will tell
> us if the file is good or, if there's something bad, we'll know where the
> problem is (in which block). The advantage of this method is that unless the
> 3 bad fragments are damaged on the same block, we may be able to recover the
> whole file.
Stat command result on all nodes:
glfs-node11.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430656 IO Block: 4096 regular file
Device: 815h/2069d Inode: 81618486372 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:40.086093439 +0300
Modify: 2018-04-06 09:37:40.056093815 +0300
Change: 2018-04-10 12:05:31.442268179 +0300
Birth: -
glfs-node12.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430656 IO Block: 4096 regular file
Device: 815h/2069d Inode: 77312455458 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:40.086927017 +0300
Modify: 2018-04-06 09:37:40.056927261 +0300
Change: 2018-04-10 12:05:31.442493868 +0300
Birth: -
glfs-node13.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 2351140352 Blocks: 2898080 IO Block: 4096 regular file
Device: 815h/2069d Inode: 77309833852 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:19:41.347484000 +0300
Modify: 2018-04-06 09:37:43.501667158 +0300
Change: 2018-04-10 12:05:31.443246800 +0300
Birth: -
glfs-node14.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430656 IO Block: 4096 regular file
Device: 815h/2069d Inode: 45097173507 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:40.085762909 +0300
Modify: 2018-04-06 09:37:40.055763093 +0300
Change: 2018-04-10 12:05:31.442878601 +0300
Birth: -
glfs-node15.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430656 IO Block: 4096 regular file
Device: 815h/2069d Inode: 47297640679 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:40.084025418 +0300
Modify: 2018-04-06 09:37:40.054025522 +0300
Change: 2018-04-10 12:05:31.443444637 +0300
Birth: -
glfs-node16.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430656 IO Block: 4096 regular file
Device: 815h/2069d Inode: 62287697396 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:40.084973842 +0300
Modify: 2018-04-06 09:37:40.054974225 +0300
Change: 2018-04-10 12:05:31.450825340 +0300
Birth: -
glfs-node17.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430656 IO Block: 4096 regular file
Device: 815h/2069d Inode: 25769816893 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:40.087142929 +0300
Modify: 2018-04-06 09:37:40.057143241 +0300
Change: 2018-04-10 12:05:31.444507483 +0300
Birth: -
glfs-node18.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430656 IO Block: 4096 regular file
Device: 815h/2069d Inode: 81645506748 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:40.085410682 +0300
Modify: 2018-04-06 09:37:40.055411031 +0300
Change: 2018-04-10 12:05:31.445081386 +0300
Birth: -
glfs-node19.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430648 IO Block: 4096 regular file
Device: 815h/2069d Inode: 55834592083 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:39.000000000 +0300
Modify: 2018-04-06 09:37:39.000000000 +0300
Change: 2018-04-10 13:43:32.589505057 +0300
Birth: -
glfs-node20.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430656 IO Block: 4096 regular file
Device: 815h/2069d Inode: 64483022647 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:40.086797717 +0300
Modify: 2018-04-06 09:37:40.056797940 +0300
Change: 2018-04-10 12:05:31.447094554 +0300
Birth: -
glfs-node21.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430648 IO Block: 4096 regular file
Device: 815h/2069d Inode: 30399803420 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:40.084872465 +0300
Modify: 2018-04-06 09:37:40.054872712 +0300
Change: 2018-04-10 12:05:31.445816033 +0300
Birth: -
glfs-node22.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430656 IO Block: 4096 regular file
Device: 815h/2069d Inode: 10740329468 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:40.087255449 +0300
Modify: 2018-04-06 09:37:40.057255732 +0300
Change: 2018-04-10 12:05:31.447168330 +0300
Birth: -
glfs-node23.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430656 IO Block: 4096 regular file
Device: 815h/2069d Inode: 73019182467 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ UNKNOWN) Gid: ( 107/ UNKNOWN)
Access: 2018-04-06 09:37:40.088930999 +0300
Modify: 2018-04-06 09:37:40.058931201 +0300
Change: 2018-04-10 12:05:31.449991404 +0300
Birth: -
glfs-node24.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 3804491264 Blocks: 7430656 IO Block: 4096 regular file
Device: 815h/2069d Inode: 219231 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ qemu) Gid: ( 107/ qemu)
Access: 2018-04-06 09:37:40.087524522 +0300
Modify: 2018-04-06 09:37:40.057524854 +0300
Change: 2018-04-10 12:05:31.448713916 +0300
Birth: -
glfs-node25.avp.ru
File: ‘/data1/bricks/brick1/vmfs/slake-test-bck-m1-d1.qcow2’
Size: 2351140352 Blocks: 2898080 IO Block: 4096 regular file
Device: 815h/2069d Inode: 55834924933 Links: 2
Access: (0770/-rwxrwx---) Uid: ( 107/ qemu) Gid: ( 107/ qemu)
Access: 2018-04-06 09:19:41.347484000 +0300
Modify: 2018-04-06 09:37:43.502748628 +0300
Change: 2018-04-10 12:05:31.449630601 +0300
Birth: -
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=f8BGycQv60&a=cc_unsubscribe
More information about the Bugs
mailing list