[Gluster-users] Advice on moving volumes/bricks to new servers

Sun Mar 1 00:02:33 UTC 2020

Hi all,

I have a 4-server system running a distributed-replicate setup, 4 x (2 + 1) = 12. Bricks are staggered across the servers. Sharding is enabled. (v info shown below)

Now, the storage is slow on the these servers and not really up to the job so we have 4 new servers with SSDs. I have to move everything over to the new servers whilst not taking down the storage.

The four old servers are running Gluster 6.4 and the new ones, 6.5.

So having read tons of docs and mailing lists, etc, I think I ought to be able to use add-brick, remove-brick to get everything moved safely like so:

# gluster volume add-brick iscsi replica 3 arbiter 1 srv{13..15}:/brick1

# gluster volume remove-brick iscsi replica 3 srv{1..3}:/brick1 start

Then once complete, do:

# gluster volume remove-brick iscsi replica 3 srv{1..3}:/brick1 commit

So I created a test volume to try this out. On the third add/remove of 4, I get a 'failed' on the remove-brick status. The rebalance log shows:

[2020-02-28 22:25:28.133902] I [dht-rebalance.c:1589:dht_migrate_file] 0-testmig
rate-dht: /linux-5.4.22/arch/arm/boot/dts/exynos4412-itop-scp-core.dtsi: attempt
ing to move from testmigrate-replicate-0 to testmigrate-replicate-2
[2020-02-28 22:25:28.144258] W [MSGID: 108015] [afr-self-heal-name.c:138:__afr_s
elfheal_name_expunge] 0-testmigrate-replicate-0: expunging file a75a83b7-2c34-40
77-b4fc-3126a9d6058a/exynos4210-smdkv310.dts (11a47b1f-2c24-4d4b-9402-9130125cf9
53) on testmigrate-client-6
[2020-02-28 22:25:28.146321] E [MSGID: 109023] [dht-rebalance.c:1707:dht_migrate_file] 0-testmigrate-dht: Migrate file failed:/linux-5.4.22/arch/arm/boot/dts/exynos4210-smdkv310.dts: lookup failed on testmigrate-replicate-0 [No such file or directory]
[2020-02-28 22:25:28.149104] E [MSGID: 109023] [dht-rebalance.c:2874:gf_defrag_migrate_single_file] 0-testmigrate-dht: migrate-data failed for /linux-5.4.22/arch/arm/boot/dts/exynos4210-smdkv310.dts [No such file or directory]

This is show for 4 files.

When I look at the FUSE-mounted volume, the file is there and correct but the file permissions of this and lots of others are screwed. Lots of dirs with d--------- permissions, lots of root:root owned files.

So any advice for how to proceed from here:

I did a force on the remove-brick as the data seemed to be in place which is fine, but now I can't do an add-brick as gluster seems to think a rebalance is taking place:

---
volume add-brick: failed: Pre Validation failed on terek-stor.amazing-internet.net. Volume name testmigrate rebalance is in progress. Please retry after completion
---

$ sudo gluster volume rebalance testmigrate status                volume rebalance: testmigrate: failed: Rebalance not started for volume testmigrate.

Thanks for any insight anyone can offer.

Ronny

$ sudo gluster volume info iscsi

Volume Name: iscsi
Type: Distributed-Replicate
Volume ID: 40ff42a7-5dee-4a98-991b-c4ba5bc50438
Status: Started
Snapshot Count: 0
Number of Bricks: 4 x (2 + 1) = 12
Transport-type: tcp
Bricks:
Brick1: ahren-stor.amazing-internet.net:/data/glusterfs/iscsi/brick1/brick
Brick2: mareth-stor.amazing-internet.net:/data/glusterfs/iscsi/brick1/brick
Brick3: terek-stor.amazing-internet.net:/data/glusterfs/iscsi/brick1a/brick (arbiter)
Brick4: walker-stor.amazing-internet.net:/data/glusterfs/iscsi/brick2/brick
Brick5: ahren-stor.amazing-internet.net:/data/glusterfs/iscsi/brick2/brick
Brick6: mareth-stor.amazing-internet.net:/data/glusterfs/iscsi/brick2a/brick (arbiter)
Brick7: terek-stor.amazing-internet.net:/data/glusterfs/iscsi/brick3/brick
Brick8: walker-stor.amazing-internet.net:/data/glusterfs/iscsi/brick3/brick
Brick9: ahren-stor.amazing-internet.net:/data/glusterfs/iscsi/brick3a/brick (arbiter)
Brick10: mareth-stor.amazing-internet.net:/data/glusterfs/iscsi/brick4/brick
Brick11: terek-stor.amazing-internet.net:/data/glusterfs/iscsi/brick4/brick
Brick12: walker-stor.amazing-internet.net:/data/glusterfs/iscsi/brick4a/brick (arbiter)
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
performance.open-behind: off
performance.readdir-ahead: off
performance.strict-o-direct: on
network.remote-dio: disable
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
features.shard-block-size: 64MB
user.cifs: off
server.allow-insecure: on
cluster.choose-local: off
auth.allow: 127.0.0.1,172.16.36.*,172.16.40.*
ssl.cipher-list: HIGH:!SSLv2
server.ssl: on
client.ssl: on
ssl.certificate-depth: 1
performance.cache-size: 1GB
client.event-threads: 4
server.event-threads: 4

-- 
Ronny Adsetts
Technical Director
Amazing Internet Ltd, London
t: +44 20 8977 8943
w: www.amazinginternet.com

Registered office: 85 Waldegrave Park, Twickenham, TW1 4TJ
Registered in England. Company No. 4042957

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: OpenPGP digital signature
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200301/0b17f134/attachment.sig>