[Gluster-users] remove-brick question

Tue Sep 17 14:47:14 UTC 2013

Thanks for your replies.

The vols seem to match the bricks ok.

FWIW, gfs-node01:/sda is the first brick; perhaps it is getting the lion's
share of the pointers?

The results of a log search and seem even more confusing.  sdb drained
rather than sda, but an error in rebalancing shows up in sdd.  I include
excerpts from scratch-rebalance, ls -l and getfattr, and bricks/sda

Does any of this suggest anything?

One failure is an apparent duplicate.  This seems to refer to the relevant
brick, and the date is correct.
[2013-09-15 19:10:07.620881] W [client3_1-fops.c:258:client3_1_mknod_cbk]
0-scratch-client-0: remote operation failed: File exists. Path:
/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118906_Qtot1500.h5.out
(00000000-0000-0000-0000-000000000000)

On the array that actually drained (mostly):
[2013-09-15 19:10:19.483040] W [client3_1-fops.c:647:client3_1_unlink_cbk]
0-scratch-client-12: remote operation failed: No such file or directory
[2013-09-15 19:10:19.483122] W [client3_1-fops.c:647:client3_1_unlink_cbk]
0-scratch-client-12: remote operation failed: No such file or directory
[2013-09-15 19:10:19.494585] W [client3_1-fops.c:258:client3_1_mknod_cbk]
0-scratch-client-12: remote operation failed: File exists. Path:
/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118429_Qtot1500.h5
(00000000-0000-0000-0000-000000000000)
[2013-09-15 19:10:19.494701] W [client3_1-fops.c:258:client3_1_mknod_cbk]
0-scratch-client-12: remote operation failed: File exists. Path:
/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118429_Qtot1500.h5
(00000000-0000-0000-0000-000000000000)

An example failure where I can trace the files is an apparent duplicate:
gfs-node01 # grep -A2 -B2 Level2_IC86.2011_data_Run00118218_Qtot1500.h5
scratch-rebalance.log
[2013-09-15 19:10:30.164409] W [client3_1-fops.c:258:client3_1_mknod_cbk]
0-scratch-client-3: remote operation failed: File exists. Path:
/nwhitehorn/vetoblast/data/Level2a_IC79_data_Run00117874_Qtot1500.h5.out
(00000000-0000-0000-0000-000000000000)
[2013-09-15 19:10:30.164473] W [client3_1-fops.c:258:client3_1_mknod_cbk]
0-scratch-client-3: remote operation failed: File exists. Path:
/nwhitehorn/vetoblast/data/Level2a_IC79_data_Run00117874_Qtot1500.h5.out
(00000000-0000-0000-0000-000000000000)
[2013-09-15 19:10:30.176606] I
[dht-common.c:956:dht_lookup_everywhere_cbk] 0-scratch-dht: deleting stale
linkfile
/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
on scratch-client-2
[2013-09-15 19:10:30.176717] I
[dht-common.c:956:dht_lookup_everywhere_cbk] 0-scratch-dht: deleting stale
linkfile
/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
on scratch-client-2
[2013-09-15 19:10:30.176856] I
[dht-common.c:956:dht_lookup_everywhere_cbk] 0-scratch-dht: deleting stale
linkfile
/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
on scratch-client-2
[2013-09-15 19:10:30.177232] W [client3_1-fops.c:647:client3_1_unlink_cbk]
0-scratch-client-2: remote operation failed: No such file or directory
[2013-09-15 19:10:30.177303] W [client3_1-fops.c:647:client3_1_unlink_cbk]
0-scratch-client-2: remote operation failed: No such file or directory
[2013-09-15 19:10:30.178101] W [client3_1-fops.c:258:client3_1_mknod_cbk]
0-scratch-client-3: remote operation failed: File exists. Path:
/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
(00000000-0000-0000-0000-000000000000)
[2013-09-15 19:10:30.178150] W [client3_1-fops.c:258:client3_1_mknod_cbk]
0-scratch-client-3: remote operation failed: File exists. Path:
/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
(00000000-0000-0000-0000-000000000000)
[2013-09-15 19:10:30.192605] W
[client3_1-fops.c:2566:client3_1_opendir_cbk] 0-scratch-client-7: remote
operation failed: No such file or directory. Path:
/nwhitehorn/vetoblast/data (00000000-0000-0000-0000-000000000000)
[2013-09-15 19:10:30.192830] W
[client3_1-fops.c:2566:client3_1_opendir_cbk] 0-scratch-client-7: remote
operation failed: No such file or directory. Path:
/nwhitehorn/vetoblast/data (00000000-0000-0000-0000-000000000000)

gfs-node01 # ls -l
/sdd/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
---------T 2 34037 40978 0 Sep 15 14:10
/sdd/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
gfs-node01 # ssh i3admin at gfs-node06 sudo ls -l
/sdb/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
-rw-r--r-- 2 34037 40978 715359 May  1 22:28
/sdb/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5

gfs-node01 #  getfattr -d -m . -e hex 
/sdd/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
# file:
sdd/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
trusted.gfid=0x11fb3ffd87be4ce3a88576466279819f
trusted.glusterfs.dht.linkto=0x736372617463682d636c69656e742d313200
gfs-node01  # ssh i3admin at gfs-node06 sudo  getfattr -d -m . -e hex 
/sdb/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
# file:
sdb/nwhitehorn/vetoblast/data/Level2_IC86.2011_data_Run00118218_Qtot1500.h5
trusted.gfid=0x11fb3ffd87be4ce3a88576466279819f

Further.
#  getfattr -d -m . -e hex  /sdd
# file: sdd
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000bffffffdd5555551
trusted.glusterfs.volume-id=0xde1fbb473e5a45dc8df804f7f73a3ecc

gfs-node01 #  getfattr -d -m . -e hex  /sdc
# file: sdc
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000aaaaaaa8bffffffc
trusted.glusterfs.volume-id=0xde1fbb473e5a45dc8df804f7f73a3ecc

gfs-node01 # getfattr -d -m . -e hex  /sdb
# file: sdb
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000000000000000000000
trusted.glusterfs.volume-id=0xde1fbb473e5a45dc8df804f7f73a3ecc

gfs-node01 #  getfattr -d -m . -e hex  /sda
# file: sda
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x0000000100000000555555546aaaaaa8
trusted.glusterfs.volume-id=0xde1fbb473e5a45dc8df804f7f73a3ecc

gfs-node01 # ssh i3admin at gfs-node04 sudo getfattr -d -m . -e hex  /sdb
# file: sdb
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x00000001000000002aaaaaaa3ffffffe
trusted.glusterfs.volume-id=0xde1fbb473e5a45dc8df804f7f73a3ecc

bricks/sda etc logs have a rather monotonous
[2013-09-16 22:23:01.723146] I [server-handshake.c:571:server_setvolume]
0-scratch-server: accepted client from
node086-11928-2013/09/16-22:22:57:696729-scratch-client-0-0 (version:
3.3.2)
[2013-09-16 22:23:01.769154] I [server.c:703:server_rpc_notify]
0-scratch-server: disconnecting connectionfrom
node086-11928-2013/09/16-22:22:57:696729-scratch-client-0-0
[2013-09-16 22:23:01.769211] I
[server-helpers.c:741:server_connection_put] 0-scratch-server: Shutting
down connection
node086-11928-2013/09/16-22:22:57:696729-scratch-client-0-0
[2013-09-16 22:23:01.769253] I
[server-helpers.c:629:server_connection_destroy] 0-scratch-server:
destroyed connection of
node086-11928-2013/09/16-22:22:57:696729-scratch-client-0-0

> On 09/17/2013 03:26 AM, james.bellinger at icecube.wisc.edu wrote:
>> I inherited a system with a wide mix of array sizes (no replication) in
>> 3.2.2, and wanted to drain data from a failing array.
>>
>> I upgraded to 3.3.2, and began a
>> gluster volume remove-brick scratch "gfs-node01:/sda" start
>>
>> After some time I got this:
>> gluster volume remove-brick scratch "gfs-node01:/sda" status
>> Node Rebalanced-files          size       scanned      failures
>> status
>>   ---------      -----------   -----------   -----------   -----------
>> ------------
>> localhost                0        0Bytes             0             0
>> not started
>> gfs-node06                0        0Bytes             0             0
>> not started
>> gfs-node03                0        0Bytes             0             0
>> not started
>> gfs-node05                0        0Bytes             0             0
>> not started
>> gfs-node01       2257394624         2.8TB       5161640        208878
>> completed
>>
>> Two things jump instantly to mind:
>> 1) The number of failures is rather large
> Can you see the rebalance logs (/var/log/scratch-rebalance.log) to
> figure out what the error messages are?
>> 2) A _different_ disk seems to have been _partially_ drained.
>> /dev/sda              2.8T  2.7T   12G 100% /sda
>> /dev/sdb              2.8T  769G  2.0T  28% /sdb
>> /dev/sdc              2.8T  2.1T  698G  75% /sdc
>> /dev/sdd              2.8T  2.2T  589G  79% /sdd
>>
>>
> I know this sounds silly, but just to be sure, is  /dev/sda actually
> mounted on "gfs-node01:sda"?
> If yes,the files that _were_ successfully rebalanced should have been
> moved from gfs-node01:sda to one of the other bricks. Is that the case?
>
>> When I mount the system it is read-only (another problem I want to fix
> Again, the mount logs could shed some information ..
> (btw a successful rebalance start/status sequence should be followed by
> the rebalance 'commit' command to ensure the volume information gets
> updated)
>
>> ASAP) so I'm pretty sure the failures aren't due to users changing the
>> system underneath me.
>>
>> Thanks for any pointers.
>>
>> James Bellinger
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>