[Gluster-devel] Gluster 9.3 not distributing data evenly across bricks
Shreyansh Shah
shreyansh.shah at alpha-grep.com
Fri Dec 1 12:37:08 UTC 2023
Hi,
We are running a gluster 9.3 volume with 47 bricks across 13 nodes. We are
noticing uneven data distribution across the bricks. Some bricks are 100%
utilised whereas some are < 85%. Due to this, data write randomly fails
with error OSError(28, 'No space left on device') even when the volume has
sufficient space on other bricks. We expect the glusterfs process to
automatically sync and maintain equal data distribution across the bricks.
This is the volume info:
root at nas-5:~# gluster v info
Volume Name: datavol
Type: Distribute
Volume ID: b4b52b4b-0ea0-4eeb-a359-5a0573d4f83a
Status: Started
Snapshot Count: 0
Number of Bricks: 47
Transport-type: tcp
Bricks:
Brick1: 10.132.2.101:/data/data
Brick2: 10.132.2.101:/data1/data
Brick3: 10.132.2.101:/data2/data
Brick4: 10.132.2.101:/data3/data
Brick5: 10.132.2.102:/data/data
Brick6: 10.132.2.102:/data1/data
Brick7: 10.132.2.102:/data2/data
Brick8: 10.132.2.102:/data3/data
Brick9: 10.132.2.103:/data/data
Brick10: 10.132.2.103:/data1/data
Brick11: 10.132.2.103:/data2/data
Brick12: 10.132.2.103:/data3/data
Brick13: 10.132.2.104:/data/data
Brick14: 10.132.2.104:/data3/data
Brick15: 10.132.2.105:/data1/data
Brick16: 10.132.2.105:/data2/data
Brick17: 10.132.2.106:/data/data
Brick18: 10.132.2.106:/data1/data
Brick19: 10.132.2.106:/data2/data
Brick20: 10.132.2.107:/data/data
Brick21: 10.132.2.107:/data1/data
Brick22: 10.132.2.107:/data2/data
Brick23: 10.132.2.108:/data/data
Brick24: 10.132.2.108:/data1/data
Brick25: 10.132.2.108:/data2/data
Brick26: 10.132.2.109:/data/data
Brick27: 10.132.2.109:/data1/data
Brick28: 10.132.2.109:/data2/data
Brick29: 10.132.2.110:/data/data
Brick30: 10.132.2.110:/data1/data
Brick31: 10.132.2.111:/data/data
Brick32: 10.132.2.111:/data1/data
Brick33: 10.132.2.111:/data2/data
Brick34: 10.132.2.112:/data/data
Brick35: 10.132.2.112:/data1/data
Brick36: 10.132.2.112:/data2/data
Brick37: 10.132.2.113:/data/data
Brick38: 10.132.2.113:/data1/data
Brick39: 10.132.2.113:/data2/data
Brick40: 10.132.2.108:/data3/data
Brick41: 10.132.2.107:/data3/data
Brick42: 10.132.2.106:/data3/data
Brick43: 10.132.2.105:/data3/data
Brick44: 10.132.2.110:/data2/data
Brick45: 10.132.2.105:/data/data
Brick46: 10.132.2.104:/data1/data
Brick47: 10.132.2.104:/data2/data
Options Reconfigured:
performance.client-io-threads: on
cluster.min-free-disk: 2%
performance.cache-refresh-timeout: 60
client.event-threads: 4
server.event-threads: 4
network.ping-timeout: 90
storage.health-check-interval: 60
storage.health-check-timeout: 60
performance.io-cache-size: 8GB
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
root at nas-5:~# glusterfs --version
glusterfs 9.3
Repository revision: git://git.gluster.org/glusterfs.git
Copyright (c) 2006-2016 Red Hat, Inc. <https://www.gluster.org/>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
It is licensed to you under your choice of the GNU Lesser
General Public License, version 3 or any later version (LGPLv3
or later), or the GNU General Public License, version 2 (GPLv2),
in all cases as published by the Free Software Foundation.
root at nas-5:~#
This is the disk usage distribution when the write failed:
*Node 1:*
/dev/bcache1 9.7T 8.5T 1.2T 88% /data
/dev/bcache3 3.9T 3.3T 563G 86% /data1
/dev/bcache0 3.9T 3.3T 560G 86% /data2
/dev/bcache2 3.9T 3.5T 395G 90% /data3
*Node 2:*
/dev/bcache0 9.7T 8.5T 1.2T 88% /data
/dev/bcache1 3.9T 3.5T 421G 90% /data1
/dev/bcache2 3.9T 3.6T 330G 92% /data2
/dev/bcache3 3.9T 3.4T 513G 87% /data3
*Node 3:*
/dev/bcache0 9.7T 8.8T 882G 92% /data
/dev/bcache1 3.9T 3.6T 335G 92% /data1
/dev/bcache2 3.9T 3.4T 532G 87% /data2
/dev/bcache3 3.9T 3.3T 564G 86% /data3
*Node 4:*
/dev/bcache0 9.7T 8.7T 982G 91% /data
/dev/bcache1 3.9T 3.5T 424G 90% /data1
/dev/bcache2 3.9T 3.4T 549G 87% /data2
/dev/bcache3 3.9T 3.6T 344G 92% /data3
*Node 5:*
/dev/bcache0 9.7T 8.5T 1.3T 88% /data
/dev/bcache1 3.9T 3.6T 288G 93% /data1
/dev/bcache2 3.9T 3.4T 470G 89% /data2
/dev/bcache3 9.9T 9.8T 101G 100% /data3
*Node 6:*
/dev/bcache0 9.7T 8.2T 1.5T 86% /data
/dev/bcache1 3.9T 3.4T 526G 87% /data1
/dev/bcache2 3.9T 3.5T 431G 90% /data2
/dev/bcache3 9.9T 8.9T 1.1T 90% /data3
*Node 7:*
/dev/bcache0 9.7T 8.9T 783G 93% /data
/dev/bcache1 3.9T 3.3T 561G 86% /data1
/dev/bcache2 3.9T 3.5T 360G 91% /data2
/dev/bcache3 9.9T 8.7T 1.2T 89% /data3
*Node 8:*
/dev/bcache0 9.7T 8.7T 994G 90% /data
/dev/bcache1 3.9T 3.3T 645G 84% /data1
/dev/bcache2 3.9T 3.4T 519G 87% /data2
/dev/bcache3 9.9T 9.0T 868G 92% /data3
*Node 9:*
/dev/bcache0 10T 8.6T 1.4T 87% /data
/dev/bcache1 8.0T 6.7T 1.4T 84% /data1
/dev/bcache2 8.0T 6.8T 1.3T 85% /data2
*Node 10:*
/dev/bcache0 10T 8.8T 1.3T 88% /data
/dev/bcache1 8.0T 6.6T 1.4T 83% /data1
/dev/bcache2 8.0T 7.0T 990G 88% /data2
*Node 11:*
/dev/bcache0 10T 8.1T 1.9T 82% /data
/dev/bcache1 10T 8.5T 1.5T 86% /data1
/dev/bcache2 10T 8.4T 1.6T 85% /data2
*Node 12:*
/dev/bcache0 10T 8.4T 1.6T 85% /data
/dev/bcache1 10T 8.4T 1.6T 85% /data1
/dev/bcache2 10T 8.2T 1.8T 83% /data2
*Node 13:*
/dev/bcache1 10T 8.7T 1.3T 88% /data1
/dev/bcache2 10T 8.8T 1.2T 88% /data2
/dev/bcache0 10T 8.6T 1.5T 86% /data
--
Regards,
Shreyansh Shah
AlphaGrep* Securities Pvt. Ltd.*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20231201/3f9e1850/attachment.html>
More information about the Gluster-devel
mailing list