[Gluster-users] cluster.min-free-disk is not working in distributed disperse volume

Thu Aug 27 12:11:02 UTC 2015

Hi Susant,

Thanks for your reply.I think, we started the re-balance with force option.
I requested a question on this mail thread regarding to run the rebalancer
daemon forcefully on a dedicated peered node instead of selecting a cluster
node automatically.

If i start rebalancer, the rebalancer daemon starts the fixlayout on all
nodes and migrate files on anyone node(always node 1 on our cluster),
The first node cpu usage is always high during rebalance compare with rest
of the cluster nodes.

To reduce the cpu usage of rebalancer datanode( node1), i peered a new
node( without disk) for re-balance and started the rebalancer. It started
again the rebalancer on same node1.

 Is there any way to run the rebalancer daemon( file migration) forcefully
on a dedicated peered node?

Regards
Backer

On Thu, Aug 27, 2015 at 3:00 PM, Susant Palai <spalai at redhat.com> wrote:

> comments inline.
> ++Ccing Pranith and Ashish to detail on disperse behaviour.
>
> ----- Original Message -----
> From: "Mohamed Pakkeer" <mdfakkeer at gmail.com>
> To: "Susant Palai" <spalai at redhat.com>, "Vijay Bellur" <vbellur at redhat.com
> >
> Cc: "Mathieu Chateau" <mathieu.chateau at lotp.fr>, "gluster-users" <
> gluster-users at gluster.org>, "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Wednesday, 26 August, 2015 2:08:02 PM
> Subject: Re: [Gluster-users] cluster.min-free-disk is not working in
> distributed disperse volume
>
>
> Hi Susant,
>
>
>
> I am monitoring the re-balance daemon data movement. I have few queries
> about this data movement.I can see the data movement on the following cases.
>
>
> 1. Source sub-volume is 90% occupied and destination sub-volume is 98%
> occupied.
> log info
> [2015-08-26 06:53:26.219674] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-qubevaultdr-dht: completed
> migration of
> /Packages/Features/MPEG/B/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_RIM_IOP_OV/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_20150401_RIM_IOP_OV/BROKEN-HORSES_EN-en_INT_REV_MPEG-reel-3-mpeg2.mxf
> from subvolume qubevaultdr-disperse-7 to qubevaultdr-disperse-10
>
> Can you mention whether you gave force option while starting rebalance? If
> force flag is set, rebalance will move the data with out considering
> whether it is moving the data from a high avail space to a low avail space.
>
>
> 2. Source sub-volume is 98% occupied and destination sub-volume is 97%
> occupied
> log info
> [2015-08-26 06:52:18.071802] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R3_MPEG_AUDIO_250514.mxf:
> attempting to move from glustertest-disperse-10 to glustertest-disperse-8
>
>
> 3. source sub-volume is 94% occupied and destination sub-volume is 15%
> occupied
> log info
> [2015-08-26 06:53:26.348804] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/ASSETMAP
> from sub-volume glustertest-disperse-26 to glustertest-disperse-56
>
>
> I can see the case 2 and 3 are valid cases because source has more data
> than destination. But Case 1 is invalid, because source sub-volume has less
> data than destination sub-volume. Also i have one more query about case 2.
> Why does the case 2 choose the destination as 94% occupied sub-volume
> instead of so many under utilized sub-volumes( 13% to 15 % sub-volumes-
> glustertest-disperse-36 to glustertest-disperse-71 ) ?
>
>
>
>
> Rebalance Log
>
>
> [2015-08-26 06:52:18.071802] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R3_MPEG_AUDIO_250514.mxf:
> attempting to move from glustertest-disperse-10 to glustertest-disperse-8
> [2015-08-26 06:52:22.633777] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of /Packages/Features/MPEG/B/Broker2_FTR_S_TE-X
> X_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER_SMK_MPEG_260514-reel-1-mpeg2.mxf
> from subvolume glustertest-disperse-9 to glustertest-disperse-34
> [2015-08-26 06:52:22.638955] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R1-2-3-4-5-6-7-8_MPEG_240514-reel-3-mpeg2.mxf:
> attempting to move from glustertest-disperse-14 to glustertest-disperse-39
> [2015-08-26 06:52:50.171304] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of /Packages/Features/MPEG/B/Broker2_FTR_S_TE-X
> X_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R3_MPEG_AUDIO_250514.mxf
> from sub-volume glustertest-disperse-10 to glustertest-disperse-8
> [2015-08-26 06:52:50.179238] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R1-2-3-4-5-6-7-8_MPEG_240514-reel-4-mpeg2.mxf:
> attempting to move from glustertest-disperse-16 to glustertest-disperse-5
> [2015-08-26 06:53:08.579988] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of /Packages/Features/MPEG/B/BrokenHorses_FTR_S
> _EN-en_INT-XX_51_HD_RIM_IOP_OV/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_20150401_RIM_IOP_OV/BROKEN-HORSES_EN-en_INT_REV_MPEG-reel-4-mpeg2.mxf
> from subvolume glustertest-disperse-2 to glustertest-disperse-23
> [2015-08-26 06:53:08.583021] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/IntegrityCheckResults.log:
> attempting to move from glustertest-disperse-18 to glustertest-disperse-0
> [2015-08-26 06:53:08.698014] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of /Packages/Features/MPEG/B/Broker2_FTR_S_TE-X
> X_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/IntegrityCheckResults.log
> from sub-volume glustertest-disperse-18 to glustertest-disperse-0
> [2015-08-26 06:53:08.703567] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R1-2-3-4-5-6-7-8_MPEG_240514-reel-5-mpeg2.mxf:
> attempting to move from glustertest-disperse-21 to glustertest-disperse-40
> [2015-08-26 06:53:26.219674] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of
> /Packages/Features/MPEG/B/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_RIM_IOP_OV/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_20150401_RIM_IOP_OV/BROKEN-HORSES_EN-en_INT_REV_MPEG-reel-3-mpeg2.mxf
> from subvolume glustertest-disperse-7 to glustertest-disperse-10
> [2015-08-26 06:53:26.226307] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/ASSETMAP:
> attempting to move from glustertest-disperse-26 to glustertest-disperse-56
> [2015-08-26 06:53:26.348804] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/ASSETMAP
> from subvolume glustertest-disperse-26 to glustertest-disperse-56
> [2015-08-26 06:53:26.354215] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R8_MPEG_AUDIO_250514.mxf:
> attempting to move from glustertest-disperse-29 to glustertest-disperse-16
>
>
>
>
> Subvolume0 to subvolume35
>
>
> root at glustertestdn001:# df -h
> Filesystem Size Used Avail Use% Mounted on Subvolume Details
> ========== ==== ==== ===== ==== =========== ==================
>
>
> /dev/sdb1 3.7T 3.6T 51G 99% /media/disk1 - glustertest-disperse-0
> /dev/sdc1 3.7T 3.3T 384G 90% /media/disk2 - glustertest-disperse-1
> /dev/sdd1 3.7T 3.6T 76G 98% /media/disk3 - glustertest-disperse-2
> /dev/sde1 3.7T 3.5T 214G 95% /media/disk4 - glustertest-disperse-3
> /dev/sdf1 3.7T 3.3T 365G 91% /media/disk5 - glustertest-disperse-4
> /dev/sdg1 3.7T 3.5T 173G 96% /media/disk6 - glustertest-disperse-5
> /dev/sdh1 3.7T 3.4T 284G 93% /media/disk7 - glustertest-disperse-6
> /dev/sdi1 3.7T 3.3T 373G 90% /media/disk8 - glustertest-disperse-7
> /dev/sdj1 3.7T 3.6T 122G 97% /media/disk9 - glustertest-disperse-8
> /dev/sdk1 3.7T 3.4T 247G 94% /media/disk10 - glustertest-disperse-9
> /dev/sdl1 3.7T 3.6T 108G 98% /media/disk11 - glustertest-disperse-10
> /dev/sdm1 3.7T 3.4T 252G 94% /media/disk12 - glustertest-disperse-11
> /dev/sdn1 3.7T 3.5T 241G 94% /media/disk13 - glustertest-disperse-12
> /dev/sdo1 3.7T 3.6T 136G 97% /media/disk14 - glustertest-disperse-13
> /dev/sdp1 3.7T 3.4T 247G 94% /media/disk15 - glustertest-disperse-14
> /dev/sdq1 3.7T 3.5T 170G 96% /media/disk16 - glustertest-disperse-15
> /dev/sdr1 3.7T 3.4T 257G 94% /media/disk17 - glustertest-disperse-16
> /dev/sds1 3.7T 3.4T 258G 94% /media/disk18 - glustertest-disperse-17
> /dev/sdt1 3.7T 3.5T 171G 96% /media/disk19 - glustertest-disperse-18
> /dev/sdu1 3.7T 3.5T 184G 96% /media/disk20 - glustertest-disperse-19
> /dev/sdv1 3.7T 3.4T 281G 93% /media/disk21 - glustertest-disperse-20
> /dev/sdw1 3.7T 3.3T 347G 91% /media/disk22 - glustertest-disperse-21
> /dev/sdx1 3.7T 3.5T 191G 95% /media/disk23 - glustertest-disperse-22
> /dev/sdy1 3.7T 3.5T 217G 95% /media/disk24 - glustertest-disperse-23
> /dev/sdz1 3.7T 3.4T 263G 93% /media/disk25 - glustertest-disperse-24
> /dev/sdaa1 3.7T 3.6T 66G 99% /media/disk26 - glustertest-disperse-25
> /dev/sdab1 3.7T 3.5T 225G 94% /media/disk27 - glustertest-disperse-26
> /dev/sdac1 3.7T 3.4T 310G 92% /media/disk28 - glustertest-disperse-27
> /dev/sdad1 3.7T 3.5T 159G 96% /media/disk29 - glustertest-disperse-28
> /dev/sdae1 3.7T 3.3T 373G 91% /media/disk30 - glustertest-disperse-29
> /dev/sdaf1 3.7T 3.4T 305G 92% /media/disk31 - glustertest-disperse-30
> /dev/sdag1 3.7T 3.5T 184G 96% /media/disk32 - glustertest-disperse-31
> /dev/sdah1 3.7T 3.5T 186G 96% /media/disk33 - glustertest-disperse-32
> /dev/sdai1 3.7T 3.3T 373G 91% /media/disk34 - glustertest-disperse-33
> /dev/sdaj1 3.7T 3.3T 361G 91% /media/disk35 - glustertest-disperse-34
> /dev/sdak1 3.7T 3.3T 375G 90% /media/disk36 - glustertest-disperse-35
>
>
>
>
>
>
> Subvolume36 to subvolume71
>
>
>
>
>
> root at glustertestdn011:/var/log/glusterfs# df -h
>
>
> Filesystem Size Used Avail Use% Mounted on SubVolume Detail
> ========== ==== ==== ===== ==== ========== ================
> /dev/sdc1 3.7T 475G 3.2T 13% /media/disk1 - glustertest-disperse-36
> /dev/sdd1 3.7T 467G 3.2T 13% /media/disk2 - glustertest-disperse-37
> /dev/sde1 3.7T 492G 3.2T 14% /media/disk3 - glustertest-disperse-38
> /dev/sdf1 3.7T 470G 3.2T 13% /media/disk4 - glustertest-disperse-39
> /dev/sdg1 3.7T 498G 3.2T 14% /media/disk5 - glustertest-disperse-40
> /dev/sdh1 3.7T 461G 3.2T 13% /media/disk6 - glustertest-disperse-41
> /dev/sdi1 3.7T 451G 3.2T 13% /media/disk7 - glustertest-disperse-42
> /dev/sdj1 3.7T 464G 3.2T 13% /media/disk8 - glustertest-disperse-43
> /dev/sdk1 3.7T 426G 3.3T 12% /media/disk9 - glustertest-disperse-44
> /dev/sdl1 3.7T 425G 3.3T 12% /media/disk10 - glustertest-disperse-45
> /dev/sdm1 3.7T 460G 3.2T 13% /media/disk11 - glustertest-disperse-46
> /dev/sdn1 3.7T 478G 3.2T 13% /media/disk12 - glustertest-disperse-47
> /dev/sdo1 3.7T 475G 3.2T 13% /media/disk13 - glustertest-disperse-48
> /dev/sdp1 3.7T 520G 3.2T 14% /media/disk14 - glustertest-disperse-49
> /dev/sdq1 3.7T 456G 3.2T 13% /media/disk15 - glustertest-disperse-50
> /dev/sdr1 3.7T 442G 3.3T 12% /media/disk16 - glustertest-disperse-51
> /dev/sds1 3.7T 494G 3.2T 14% /media/disk17 - glustertest-disperse-52
> /dev/sdt1 3.7T 495G 3.2T 14% /media/disk18 - glustertest-disperse-53
> /dev/sdu1 3.7T 426G 3.3T 12% /media/disk19 - glustertest-disperse-54
> /dev/sdv1 3.7T 471G 3.2T 13% /media/disk20 - glustertest-disperse-55
> /dev/sdw1 3.7T 531G 3.2T 15% /media/disk21 - glustertest-disperse-56
> /dev/sdx1 3.7T 576G 3.1T 16% /media/disk22 - glustertest-disperse-57
> /dev/sdy1 3.7T 411G 3.3T 12% /media/disk23 - glustertest-disperse-58
> /dev/sdz1 3.7T 455G 3.2T 13% /media/disk24 - glustertest-disperse-59
> /dev/sdaa1 3.7T 427G 3.3T 12% /media/disk25 - glustertest-disperse-60
> /dev/sdab1 3.7T 442G 3.3T 12% /media/disk26 - glustertest-disperse-61
> /dev/sdac1 3.7T 449G 3.2T 13% /media/disk27 - glustertest-disperse-62
> /dev/sdad1 3.7T 446G 3.3T 12% /media/disk28 - glustertest-disperse-63
> /dev/sdae1 3.7T 412G 3.3T 12% /media/disk29 - glustertest-disperse-64
> /dev/sdaf1 3.7T 480G 3.2T 13% /media/disk30 - glustertest-disperse-65
> /dev/sdag1 3.7T 485G 3.2T 14% /media/disk31 - glustertest-disperse-66
> /dev/sdah1 3.7T 453G 3.2T 13% /media/disk32 - glustertest-disperse-67
> /dev/sdai1 3.7T 410G 3.3T 12% /media/disk33 - glustertest-disperse-68
> /dev/sdaj1 3.7T 457G 3.2T 13% /media/disk34 - glustertest-disperse-69
> /dev/sdak1 3.7T 444G 3.3T 12% /media/disk35 - glustertest-disperse-70
> /dev/sdal1 3.7T 484G 3.2T 13% /media/disk36 - glustertest-disperse-71
>
>
>
>
> Is this the normal behavior of DHT in distributed disperse volume ?
>
> Requesting Pranith and Ashish to comment on this.
>
> Thanks
> Backer
>
>
>
>
>
>
>
> On Tue, Aug 25, 2015 at 12:36 PM, Susant Palai < spalai at redhat.com >
> wrote:
>
>
> Mohamed,
> Will investigate in to weighted rebalance behavior.
>
> Susant
>
> ----- Original Message -----
> From: "Mohamed Pakkeer" < mdfakkeer at gmail.com >
> To: "Susant Palai" < spalai at redhat.com >
> Cc: "Mathieu Chateau" < mathieu.chateau at lotp.fr >, "gluster-users" <
> gluster-users at gluster.org >, "Gluster Devel" < gluster-devel at gluster.org >
> Sent: Tuesday, 25 August, 2015 9:40:01 AM
> Subject: Re: [Gluster-users] cluster.min-free-disk is not working in
> distributed disperse volume
>
>
>
>
> Hi Sasant,
> We have created the disperse volume across nodes. We stopped all the
> upload operations and started the rebalance last night.After overnight
> re-balance, some harddisk is occupied 100% and some disks have 13% disk
> space.
>
>
> disk1 belongs to disperse-set-0 ..... disk36 belongs to disperse-set-35
>
>
> df -h result of one data node
>
>
>
> /dev/sdb1 3.7T 3.7T 545M 100% /media/disk1
> /dev/sdc1 3.7T 3.2T 496G 87% /media/disk2
> /dev/sdd1 3.7T 3.7T 30G 100% /media/disk3
> /dev/sde1 3.7T 3.5T 173G 96% /media/disk4
> /dev/sdf1 3.7T 3.2T 458G 88% /media/disk5
> /dev/sdg1 3.7T 3.5T 143G 97% /media/disk6
> /dev/sdh1 3.7T 3.5T 220G 95% /media/disk7
> /dev/sdi1 3.7T 3.3T 415G 89% /media/disk8
> /dev/sdj1 3.7T 3.6T 72G 99% /media/disk9
> /dev/sdk1 3.7T 3.5T 186G 96% /media/disk10
> /dev/sdl1 3.7T 3.6T 65G 99% /media/disk11
> /dev/sdm1 3.7T 3.5T 195G 95% /media/disk12
> /dev/sdn1 3.7T 3.5T 199G 95% /media/disk13
> /dev/sdo1 3.7T 3.6T 78G 98% /media/disk14
> /dev/sdp1 3.7T 3.5T 200G 95% /media/disk15
> /dev/sdq1 3.7T 3.6T 119G 97% /media/disk16
> /dev/sdr1 3.7T 3.5T 206G 95% /media/disk17
> /dev/sds1 3.7T 3.5T 193G 95% /media/disk18
> /dev/sdt1 3.7T 3.6T 131G 97% /media/disk19
> /dev/sdu1 3.7T 3.5T 141G 97% /media/disk20
> /dev/sdv1 3.7T 3.5T 243G 94% /media/disk21
> /dev/sdw1 3.7T 3.4T 299G 92% /media/disk22
> /dev/sdx1 3.7T 3.5T 163G 96% /media/disk23
> /dev/sdy1 3.7T 3.5T 168G 96% /media/disk24
> /dev/sdz1 3.7T 3.5T 219G 95% /media/disk25
> /dev/sdaa1 3.7T 3.7T 37G 100% /media/disk26
> /dev/sdab1 3.7T 3.5T 172G 96% /media/disk27
> /dev/sdac1 3.7T 3.4T 276G 93% /media/disk28
> /dev/sdad1 3.7T 3.6T 108G 98% /media/disk29
> /dev/sdae1 3.7T 3.3T 399G 90% /media/disk30
> /dev/sdaf1 3.7T 3.5T 240G 94% /media/disk31
> /dev/sdag1 3.7T 3.6T 122G 97% /media/disk32
> /dev/sdah1 3.7T 3.5T 147G 97% /media/disk33
> /dev/sdai1 3.7T 3.4T 342G 91% /media/disk34
> /dev/sdaj1 3.7T 3.4T 288G 93% /media/disk35
> /dev/sdak1 3.7T 3.4T 342G 91% /media/disk36
>
>
> disk1 belongs to disperse-set-0. Rebalancer logs shows, still rebalancer
> is trying to fill the disperse-set-0 after filling to 100%
>
>
>
>
> [2015-08-24 19:52:53.036622] E [MSGID: 109023]
> [dht-rebalance.c:672:__dht_check_free_space] 0-glustertest-dht: data
> movement attempted from node (glustertest-disperse-7) to node
> (glustertest-disperse-0) which does not have required free space for
> (/Packages/Features/MPEG/A/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_RIC_OV/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_20110521_RIC_OV/AMI-NEBO-C_R3_AUDIO_190511.mxf)
>
>
> [2015-08-24 19:52:53.042026] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/A/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_RIC_OV/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_20110521_RIC_OV/AMINEBO-CHALLANGE_BEN_R1-2-3-4-5-6_MPEG_200511-reel-5-mpeg2.mxf:
> attempting to move from glustertest-disperse-13 to glustertest-disperse-0
>
>
> I think, cluster.weighted-rebalance and cluster.min-free-dis k have bugs
> for re-balancing the data, based on weight and disk free space.
>
>
>
>
> Thanks
> Backer
>
>
>
>
> On Mon, Aug 24, 2015 at 4:28 PM, Mohamed Pakkeer < mdfakkeer at gmail.com >
> wrote:
>
>
>
> Hi Susant,
>
>
> Thanks for your quick reply. We are not updating any files. Actually we
> are archiving video files on this cluster. I think there is a bug in
> cluster.min-free-disk.
>
>
> Also i would like to know about rebalance the cluster. Currently we have
> 20 nodes and 10 nodes hard disks are almost full . So we need to rebalance
> the data. If i run the rebalancer, it starts on first node(node1) and
> starts the migration process. The first node cpu usage is always high
> during rebalance compare with rest of the cluster nodes.To reduce the cpu
> usage of rebalancer datanode( node1), i peer a new node( without disk) for
> rebalance and start the rebalancer. It starts again the rebalancer on same
> node1. How can we run a rebalancer on a dedicated node?
>
>
> Also we are facing memory leaks in fixlayout and heal full operations.
>
>
> Regards
> Backer
>
>
>
>
> On Mon, Aug 24, 2015 at 2:57 PM, Susant Palai < spalai at redhat.com > wrote:
>
>
> Hi,
> Cluster.min-free-disk controls new file creation on the bricks. If you
> happen to write to the existing files on the brick and that is leading to
> brick getting full, then most probably you should run a rebalance.
>
> Regards,
> Susant
>
>
>
> ----- Original Message -----
> From: "Mathieu Chateau" < mathieu.chateau at lotp.fr >
> To: "Mohamed Pakkeer" < mdfakkeer at gmail.com >
> Cc: "gluster-users" < gluster-users at gluster.org >, "Gluster Devel" <
> gluster-devel at gluster.org >
> Sent: Monday, 24 August, 2015 2:47:00 PM
> Subject: Re: [Gluster-users] cluster.min-free-disk is not working in
> distributed disperse volume
>
>
>
>
> 720 brick! Respect !
> Le 24 août 2015 09:48, "Mohamed Pakkeer" < mdfakkeer at gmail.com > a écrit :
>
>
>
> Hi,
>
>
> I have a cluster of 720 bricks, all bricks are 4TB in size. I have change
> the cluster.min-free-disk default value 10% to 3%. So all the disks should
> have 3% minimum disk space free. But some cluster disks are getting full
> now. Is there any additional configuration for keeping some percentage of
> disk space kept free?
>
>
>
>
>
> Volume Name: glustertest
> Type: Distributed-Disperse
> Volume ID: 2b575b5c-df2e-449c-abb9-c56cec27e609
> Status: Started
> Number of Bricks: 72 x (8 + 2) = 720
> Transport-type: tcp
>
>
>
>
>
> Options Reconfigured:
> features.default-soft-limit: 95%
> cluster.min-free-disk: 3%
> performance.readdir-ahead: on
>
>
> df -h of one node
>
>
>
> /dev/sdb1 3.7T 3.6T 132G 97% /media/disk1
> /dev/sdc1 3.7T 3.2T 479G 88% /media/disk2
> /dev/sdd1 3.7T 3.6T 109G 98% /media/disk3
>
>
> Any help will be greatly appreciated.
>
>
>
>
> Regards
> Backer
>
>
>
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
>
>
>
>
>
> --
>
> Thanks & Regards
> K. Mohamed Pakkeer
> Mobile- 0091-8754410114
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150827/eed21c97/attachment.html>