[Gluster-users] Missing files after add new bricks and remove old ones - how to restore files
Marco Lorenzo Crociani
marcoc at prismatelecomtesting.com
Thu Nov 5 16:27:12 UTC 2015
Any news?
glusterfs version is 3.7.5
On 30/10/2015 17:51, Marco Lorenzo Crociani wrote:
> Hi Susant,
> here the stats:
>
> [root at s20 brick1]# stat .* *
> File: `.'
> Size: 78 Blocks: 0 IO Block: 4096 directory
> Device: 811h/2065d Inode: 2481712637 Links: 7
> Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-30 15:26:57.565475699 +0100
> Modify: 2015-08-04 12:30:56.604846056 +0200
> Change: 2015-10-27 14:21:12.981420157 +0100
> File: `..'
> Size: 50 Blocks: 0 IO Block: 4096 directory
> Device: 811h/2065d Inode: 495824 Links: 6
> Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-27 11:27:36.956230769 +0100
> Modify: 2015-08-04 11:25:27.893410342 +0200
> Change: 2015-08-04 11:25:27.893410342 +0200
> File: `.glusterfs'
> Size: 8192 Blocks: 24 IO Block: 4096 directory
> Device: 811h/2065d Inode: 2481712643 Links: 261
> Access: (0600/drw-------) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-30 15:31:14.988775452 +0100
> Modify: 2015-08-04 12:31:40.803715075 +0200
> Change: 2015-08-04 12:31:40.803715075 +0200
> File: `.trashcan'
> Size: 24 Blocks: 0 IO Block: 4096 directory
> Device: 811h/2065d Inode: 495865 Links: 3
> Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 18:32:17.369070847 +0100
> Modify: 2015-08-04 11:36:11.357529000 +0200
> Change: 2015-10-26 18:32:17.368070850 +0100
> File: `lost+found'
> Size: 6 Blocks: 0 IO Block: 4096 directory
> Device: 811h/2065d Inode: 2481712624 Links: 2
> Access: (0700/drwx------) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 18:55:08.274323554 +0100
> Modify: 2014-01-18 21:48:37.000000000 +0100
> Change: 2015-10-26 18:55:08.259323594 +0100
> File: `rh'
> Size: 6 Blocks: 0 IO Block: 4096 directory
> Device: 811h/2065d Inode: 495961 Links: 2
> Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 14:02:02.294771698 +0100
> Modify: 2015-03-26 13:22:19.000000000 +0100
> Change: 2015-10-26 18:32:17.384070805 +0100
> File: `zimbra'
> Size: 4096 Blocks: 8 IO Block: 4096 directory
> Device: 811h/2065d Inode: 495969 Links: 50
> Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-30 15:27:44.899346957 +0100
> Modify: 2015-10-26 18:32:17.733069841 +0100
> Change: 2015-10-26 18:32:17.733069841 +0100
>
>
>
> [root at s21 brick2]# stat .* *
> File: `.'
> Size: 78 Blocks: 0 IO Block: 4096 directory
> Device: 811h/2065d Inode: 501309 Links: 7
> Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-30 17:45:23.983929018 +0100
> Modify: 2015-08-04 12:30:56.602392330 +0200
> Change: 2015-10-26 18:32:17.327779305 +0100
> File: `..'
> Size: 50 Blocks: 0 IO Block: 4096 directory
> Device: 811h/2065d Inode: 2484780736 Links: 6
> Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-30 17:45:20.800922878 +0100
> Modify: 2015-08-04 11:25:27.942732803 +0200
> Change: 2015-08-04 11:25:27.942732803 +0200
> File: `.glusterfs'
> Size: 8192 Blocks: 24 IO Block: 4096 directory
> Device: 811h/2065d Inode: 501323 Links: 261
> Access: (0600/drw-------) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-08-04 11:36:13.776967886 +0200
> Modify: 2015-08-04 12:31:40.801477366 +0200
> Change: 2015-08-04 12:31:40.801477366 +0200
> File: `.trashcan'
> Size: 24 Blocks: 0 IO Block: 4096 directory
> Device: 811h/2065d Inode: 2484780773 Links: 3
> Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 18:32:17.324779299 +0100
> Modify: 2015-08-04 11:36:11.357529000 +0200
> Change: 2015-10-26 18:32:17.368779386 +0100
> File: `lost+found'
> Size: 6 Blocks: 0 IO Block: 4096 directory
> Device: 811h/2065d Inode: 501268 Links: 2
> Access: (0700/drwx------) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 18:32:17.371779392 +0100
> Modify: 2014-01-18 21:48:37.000000000 +0100
> Change: 2015-10-26 18:55:08.260516194 +0100
> File: `rh'
> Size: 6 Blocks: 0 IO Block: 4096 directory
> Device: 811h/2065d Inode: 2484780842 Links: 2
> Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 18:32:17.386779422 +0100
> Modify: 2015-03-26 13:22:19.000000000 +0100
> Change: 2015-10-26 18:32:17.384779418 +0100
> File: `zimbra'
> Size: 4096 Blocks: 8 IO Block: 4096 directory
> Device: 811h/2065d Inode: 2484780856 Links: 50
> Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 18:34:10.300004939 +0100
> Modify: 2015-10-26 18:32:17.733780116 +0100
> Change: 2015-10-26 18:32:17.733780116 +0100
>
>
>
>
>
>
> [root at s20 brick1]# stat zimbra/jdk-1.7.0_45/db/bin/*
> File: `zimbra/jdk-1.7.0_45/db/bin/dblook'
> Size: 5740 Blocks: 16 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 509260 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.509803807 +0100
> Modify: 2012-11-02 12:23:32.000000000 +0100
> Change: 2015-10-26 15:31:30.510803804 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/derby_common.bat'
> Size: 2426 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 509263 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.510803804 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.511803801 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/ij'
> Size: 5876 Blocks: 16 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 509264 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.512803798 +0100
> Modify: 2012-11-02 12:23:32.000000000 +0100
> Change: 2015-10-26 15:31:30.512803798 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/NetworkServerControl'
> Size: 5801 Blocks: 16 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 509257 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.507803811 +0100
> Modify: 2012-11-02 12:23:32.000000000 +0100
> Change: 2015-10-26 15:31:30.508803809 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/setEmbeddedCP.bat'
> Size: 1278 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 509269 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.513803795 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.514803792 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkClientCP.bat'
> Size: 1284 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 509272 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.514803792 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.515803790 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkServerCP'
> Size: 1075 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 509273 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.516803788 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.517803786 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer'
> Size: 5807 Blocks: 16 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 509276 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.517803786 +0100
> Modify: 2012-11-02 12:23:32.000000000 +0100
> Change: 2015-10-26 15:31:30.518803783 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer.bat'
> Size: 1397 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 509278 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.519803780 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.520803777 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/sysinfo.bat'
> Size: 1389 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 509284 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-30 17:47:26.048360639 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.521803774 +0100
>
>
> [root at s21 brick2]# stat zimbra/jdk-1.7.0_45/db/bin/*
> File: `zimbra/jdk-1.7.0_45/db/bin/dblook'
> Size: 5740 Blocks: 16 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 2484798182 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.508105065 +0100
> Modify: 2012-11-02 12:23:32.000000000 +0100
> Change: 2015-10-26 15:31:30.509105067 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/derby_common.bat'
> Size: 2426 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 2484798185 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.510105069 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.511105071 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/ij'
> Size: 5876 Blocks: 16 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 2484798186 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.511105071 +0100
> Modify: 2012-11-02 12:23:32.000000000 +0100
> Change: 2015-10-26 15:31:30.512105073 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/NetworkServerControl'
> Size: 5801 Blocks: 16 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 2484798179 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.507105063 +0100
> Modify: 2012-11-02 12:23:32.000000000 +0100
> Change: 2015-10-26 15:31:30.508105065 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/setEmbeddedCP.bat'
> Size: 1278 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 2484798191 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.513105075 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.513105075 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkClientCP.bat'
> Size: 1284 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 2484798194 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.514105077 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.515105079 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/setNetworkServerCP'
> Size: 1075 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 2484798195 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.515105079 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.516105081 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer'
> Size: 5807 Blocks: 16 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 2484798198 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.517105083 +0100
> Modify: 2012-11-02 12:23:32.000000000 +0100
> Change: 2015-10-26 15:31:30.518105085 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/startNetworkServer.bat'
> Size: 1397 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 2484798200 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.519105087 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.519105087 +0100
> File: `zimbra/jdk-1.7.0_45/db/bin/sysinfo.bat'
> Size: 1389 Blocks: 8 IO Block: 4096 regular file
> Device: 811h/2065d Inode: 2484798206 Links: 2
> Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
> Access: 2015-10-26 15:31:30.520105089 +0100
> Modify: 2012-11-01 10:49:22.000000000 +0100
> Change: 2015-10-26 15:31:30.521105091 +0100
>
>
> Thanks,
>
> Marco
>
>
>
> On 30/10/2015 08:22, Susant Palai wrote:
>> Hi Marco,
>> Can you send the stat of the files from the removed-brick?
>>
>> Susant
>>
>> ----- Original Message -----
>> From: "Marco Lorenzo Crociani" <marcoc at prismatelecomtesting.com>
>> To: gluster-users at gluster.org
>> Sent: Tuesday, 27 October, 2015 6:58:51 PM
>> Subject: [Gluster-users] Missing files after add new bricks and
>> remove old ones - how to restore files
>>
>> Hi,
>> we had a 2 node glusterfs cluster.
>> we had a 2x2 Distributed-Replicate volume on it.
>>
>> It was:
>> Brick1: s20gfs.ovirt:/gluster/VOL/brick1
>> Brick2: s21gfs.ovirt:/gluster/VOL/brick2
>> Brick3: s20gfs.ovirt:/gluster/VOL/brick3
>> Brick4: s21gfs.ovirt:/gluster/VOL/brick4
>>
>> We added more nodes to the cluster. So I wanted to redistribute the
>> bricks on the nodes.
>> I added 2 new bricks to the volume.
>>
>> gluster volume add-brick VOL s22gfs.ovirt:/gluster/VOL/brick2
>> x`s23gfs.ovirt:/gluster/VOL/brick3
>>
>> then I removed 2 old bricks
>> gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1
>> s21gfs.ovirt:/gluster/VOL/brick2 start
>>
>> checked the status
>> gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1
>> s21gfs.ovirt:/gluster/VOL/brick2 status
>>
>> when it was completed and I saw data on the new bricks I run:
>> gluster volume remove-brick VOL s20gfs.ovirt:/gluster/VOL/brick1
>> s21gfs.ovirt:/gluster/VOL/brick2 commit
>>
>> Result:
>> some files missing from the volume. Those file are still on the removed
>> bricks.
>>
>> First question. Have I followed a wrong procedure?
>>
>> Second question. How to restore those files? I can't re-add back those
>> brick to the volume because it tolds me:
>> volume add-brick: failed: /gluster/VOL/brick1 is already part of a
>> volume
>> If I use the force option I get the old files or they get erased?
>> Should I rsync from the "unmounted" brick to the "mounted" volume or to
>> an "umounted" brick that is part of the volume?
>>
>> Regards,
>>
>
>
--
Marco Crociani
Prisma Telecom Testing S.r.l.
via Petrocchi, 4 20127 MILANO ITALY
Phone: +39 02 26113507
Fax: +39 02 26113597
e-mail: marcoc at prismatelecomtesting.com
web: http://www.prismatelecomtesting.com
Questa email (e I suoi allegati) costituisce informazione riservata e confidenziale e può essere soggetto a legal privilege. Può essere utilizzata esclusivamente dai suoi destinatari legittimi. Se avete ricevuto questa email per errore, siete pregati di informarne immediatamente il mittente e quindi cancellarla. A meno che non siate stati a ciò espressamente autorizzati, la diffusione o la riproduzione di questa email o del suo contenuto non sono consentiti.
Salvo che questa email sia espressamente qualificata come offerta o accettazione contrattuale, il mittente non intende con questa email dare vita ad un vincolo giuridico e questa email non può essere interpretata quale offerta o accettazione che possa dare vita ad un contratto. Qualsiasi opinione manifestata in questa email è un'opinione personale del mittente, salvo che il mittente dichiari espressamente che si tratti di un'opinione di Prisma Engineering.
*******************************************************************************
This e-mail (including any attachments) is private and confidential, and may be privileged. It is for the exclusive use of the intended recipient(s). If you have received this email in error, please inform the sender immediately and then delete this email. Unless you have been given specific permission to do so, please do not distribute or copy this email or its contents.
Unless the text of this email specifically states that it is a contractual offer or acceptance, the sender does not intend to create a legal relationship and this email shall not constitute an offer or acceptance which could give rise to a contract. Any views expressed in this communication are those of the individual sender, except where the sender specifically states them to be the views of Prisma Engineering.
More information about the Gluster-users
mailing list