[Gluster-users] cluster.min-free-disk is not working in distributed disperse volume
Mohamed Pakkeer
mdfakkeer at gmail.com
Fri Aug 28 10:27:37 UTC 2015
Hi Susant,
Thanks for your reply.
"Can you mention whether you gave force option while starting rebalance? If
force flag is set, rebalance will move the data with out considering
whether it is moving the data from a high avail space to a low avail space."
we are able to reproduce the same issue after stopping the rebalance with
force option and starting rebalance without force option.
source sub-volume is occupied 91% and destination sub-volume is occupied
93%( higher space disk to lower space disk)
*Rebalance Log:*
[2015-08-28 10:13:05.130492] I [dht-rebalance.c:1002:dht_migrate_file]
0-qubevaultdr-dht:
/Packages/Features/MPEG/A/AlaaElaa_FTR_S_TE-XX_IN-UA_51_HD_PIX_RIH_IOP_OV/AlaaElaa_FTR_S_TE-XX_IN-UA_51_HD_PIX_20141124_RIH_IOP_OV/ALA-ELA_R5_AUDIO_241114.mxf:
attempting to move from qubevaultdr-disperse-34 to qubevaultdr-disperse-31
/dev/sdb1 3.7T 3.4T 310G 92% /media/disk1 - disperse0
/dev/sdc1 3.7T 3.3T 371G 91% /media/disk2
/dev/sdd1 3.7T 3.5T 224G 95% /media/disk3
/dev/sde1 3.7T 3.4T 301G 92% /media/disk4
/dev/sdf1 3.7T 3.3T 356G 91% /media/disk5
/dev/sdg1 3.7T 3.5T 242G 94% /media/disk6
/dev/sdh1 3.7T 3.4T 335G 92% /media/disk7
/dev/sdi1 3.7T 3.3T 356G 91% /media/disk8
/dev/sdj1 3.7T 3.4T 272G 93% /media/disk9
/dev/sdk1 3.7T 3.4T 302G 92% /media/disk10
/dev/sdl1 3.7T 3.4T 246G 94% /media/disk11
/dev/sdm1 3.7T 3.4T 330G 92% /media/disk12
/dev/sdn1 3.7T 3.4T 339G 91% /media/disk13
/dev/sdo1 3.7T 3.4T 266G 93% /media/disk14 -
/dev/sdp1 3.7T 3.4T 342G 91% /media/disk15 -disperse14
/dev/sdq1 3.7T 3.4T 267G 93% /media/disk16
/dev/sdr1 3.7T 3.3T 358G 91% /media/disk17
/dev/sds1 3.7T 3.3T 360G 91% /media/disk18
/dev/sdt1 3.7T 3.4T 259G 94% /media/disk19
/dev/sdu1 3.7T 3.4T 313G 92% /media/disk20
/dev/sdv1 3.7T 3.3T 364G 91% /media/disk21
/dev/sdw1 3.7T 3.3T 367G 91% /media/disk22
/dev/sdx1 3.7T 3.4T 291G 93% /media/disk23
/dev/sdy1 3.7T 3.4T 302G 92% /media/disk24
/dev/sdz1 3.7T 3.3T 350G 91% /media/disk25
/dev/sdaa1 3.7T 3.5T 209G 95% /media/disk26
/dev/sdab1 3.7T 3.4T 333G 92% /media/disk27
/dev/sdac1 3.7T 3.3T 374G 90% /media/disk28
/dev/sdad1 3.7T 3.4T 318G 92% /media/disk29
/dev/sdae1 3.7T 3.3T 371G 91% /media/disk30 - disperse-29
/dev/sdaf1 3.7T 3.3T 371G 91% /media/disk31 - disperse-30
*/dev/sdag1 3.7T 3.4T 261G 93% /media/disk32* - disperse-31
*=============>
destination*
/dev/sdah1 3.7T 3.4T 273G 93% /media/disk33 - disperse-32
/dev/sdai1 3.7T 3.3T 370G 91% /media/disk34 - disperse-33
*/dev/sdaj1 3.7T 3.3T 365G 91% /media/disk35* - disperse-34
*=============>
source*
/dev/sdak1 3.7T 3.3T 366G 91% /media/disk36 - disperse-35
Thanks
Backer
On Thu, Aug 27, 2015 at 6:20 PM, Susant Palai <spalai at redhat.com> wrote:
> Comments inline.
>
> ----- Original Message -----
> From: "Mohamed Pakkeer" <mdfakkeer at gmail.com>
> To: "Susant Palai" <spalai at redhat.com>
> Cc: "Mathieu Chateau" <mathieu.chateau at lotp.fr>, "gluster-users" <
> gluster-users at gluster.org>, "Gluster Devel" <gluster-devel at gluster.org>,
> "Vijay Bellur" <vbellur at redhat.com>, "Pranith Kumar Karampuri" <
> pkarampu at redhat.com>, "Ashish Pandey" <aspandey at redhat.com>
> Sent: Thursday, 27 August, 2015 5:41:02 PM
> Subject: Re: [Gluster-users] cluster.min-free-disk is not working in
> distributed disperse volume
>
>
> Hi Susant,
>
>
> Thanks for your reply.I think, we started the re-balance with force
> option. I requested a question on this mail thread regarding to run the
> rebalancer daemon forcefully on a dedicated peered node instead of
> selecting a cluster node automatically.
> >>> Currently there is no such feature to start rebalance on a dedicated
> node.
>
> If i start rebalancer, the rebalancer daemon starts the fixlayout on all
> nodes and migrate files on anyone node(always node 1 on our cluster),
> The first node cpu usage is always high during rebalance compare with rest
> of the cluster nodes.
>
>
> To reduce the cpu usage of rebalancer datanode( node1), i peered a new
> node( without disk) for re-balance and started the rebalancer. It started
> again the rebalancer on same node1.
>
>
> Is there any way to run the rebalancer daemon( file migration) forcefully
> on a dedicated peered node?
>
>
>
>
>
> Regards
> Backer
>
>
>
>
>
>
> On Thu, Aug 27, 2015 at 3:00 PM, Susant Palai < spalai at redhat.com > wrote:
>
>
> comments inline.
> ++Ccing Pranith and Ashish to detail on disperse behaviour.
>
> ----- Original Message -----
> From: "Mohamed Pakkeer" < mdfakkeer at gmail.com >
> To: "Susant Palai" < spalai at redhat.com >, "Vijay Bellur" <
> vbellur at redhat.com >
> Cc: "Mathieu Chateau" < mathieu.chateau at lotp.fr >, "gluster-users" <
> gluster-users at gluster.org >, "Gluster Devel" < gluster-devel at gluster.org >
> Sent: Wednesday, 26 August, 2015 2:08:02 PM
> Subject: Re: [Gluster-users] cluster.min-free-disk is not working in
> distributed disperse volume
>
>
> Hi Susant,
>
>
>
> I am monitoring the re-balance daemon data movement. I have few queries
> about this data movement.I can see the data movement on the following cases.
>
>
> 1. Source sub-volume is 90% occupied and destination sub-volume is 98%
> occupied.
> log info
> [2015-08-26 06:53:26.219674] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-qubevaultdr-dht: completed
> migration of
> /Packages/Features/MPEG/B/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_RIM_IOP_OV/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_20150401_RIM_IOP_OV/BROKEN-HORSES_EN-en_INT_REV_MPEG-reel-3-mpeg2.mxf
> from subvolume qubevaultdr-disperse-7 to qubevaultdr-disperse-10
>
> Can you mention whether you gave force option while starting rebalance? If
> force flag is set, rebalance will move the data with out considering
> whether it is moving the data from a high avail space to a low avail space.
>
>
> 2. Source sub-volume is 98% occupied and destination sub-volume is 97%
> occupied
> log info
> [2015-08-26 06:52:18.071802] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R3_MPEG_AUDIO_250514.mxf:
> attempting to move from glustertest-disperse-10 to glustertest-disperse-8
>
>
> 3. source sub-volume is 94% occupied and destination sub-volume is 15%
> occupied
> log info
> [2015-08-26 06:53:26.348804] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/ASSETMAP
> from sub-volume glustertest-disperse-26 to glustertest-disperse-56
>
>
> I can see the case 2 and 3 are valid cases because source has more data
> than destination. But Case 1 is invalid, because source sub-volume has less
> data than destination sub-volume. Also i have one more query about case 2.
> Why does the case 2 choose the destination as 94% occupied sub-volume
> instead of so many under utilized sub-volumes( 13% to 15 % sub-volumes-
> glustertest-disperse-36 to glustertest-disperse-71 ) ?
>
>
>
>
>
>
> Rebalance Log
>
>
> [2015-08-26 06:52:18.071802] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R3_MPEG_AUDIO_250514.mxf:
> attempting to move from glustertest-disperse-10 to glustertest-disperse-8
> [2015-08-26 06:52:22.633777] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of /Packages/Features/MPEG/B/Broker2_FTR_S_TE-X
> X_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER_SMK_MPEG_260514-reel-1-mpeg2.mxf
> from subvolume glustertest-disperse-9 to glustertest-disperse-34
> [2015-08-26 06:52:22.638955] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R1-2-3-4-5-6-7-8_MPEG_240514-reel-3-mpeg2.mxf:
> attempting to move from glustertest-disperse-14 to glustertest-disperse-39
> [2015-08-26 06:52:50.171304] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of /Packages/Features/MPEG/B/Broker2_FTR_S_TE-X
> X_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R3_MPEG_AUDIO_250514.mxf
> from sub-volume glustertest-disperse-10 to glustertest-disperse-8
> [2015-08-26 06:52:50.179238] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R1-2-3-4-5-6-7-8_MPEG_240514-reel-4-mpeg2.mxf:
> attempting to move from glustertest-disperse-16 to glustertest-disperse-5
> [2015-08-26 06:53:08.579988] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of /Packages/Features/MPEG/B/BrokenHorses_FTR_S
> _EN-en_INT-XX_51_HD_RIM_IOP_OV/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_20150401_RIM_IOP_OV/BROKEN-HORSES_EN-en_INT_REV_MPEG-reel-4-mpeg2.mxf
> from subvolume glustertest-disperse-2 to glustertest-disperse-23
> [2015-08-26 06:53:08.583021] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/IntegrityCheckResults.log:
> attempting to move from glustertest-disperse-18 to glustertest-disperse-0
> [2015-08-26 06:53:08.698014] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of /Packages/Features/MPEG/B/Broker2_FTR_S_TE-X
> X_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/IntegrityCheckResults.log
> from sub-volume glustertest-disperse-18 to glustertest-disperse-0
> [2015-08-26 06:53:08.703567] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_
> S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R1-2-3-4-5-6-7-8_MPEG_240514-reel-5-mpeg2.mxf:
> attempting to move from glustertest-disperse-21 to glustertest-disperse-40
> [2015-08-26 06:53:26.219674] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of
> /Packages/Features/MPEG/B/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_RIM_IOP_OV/BrokenHorses_FTR_S_EN-en_INT-XX_51_HD_20150401_RIM_IOP_OV/BROKEN-HORSES_EN-en_INT_REV_MPEG-reel-3-mpeg2.mxf
> from subvolume glustertest-disperse-7 to glustertest-disperse-10
> [2015-08-26 06:53:26.226307] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/ASSETMAP:
> attempting to move from glustertest-disperse-26 to glustertest-disperse-56
> [2015-08-26 06:53:26.348804] I [MSGID: 109022]
> [dht-rebalance.c:1282:dht_migrate_file] 0-glustertest-dht: completed
> migration of
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/ASSETMAP
> from subvolume glustertest-disperse-26 to glustertest-disperse-56
> [2015-08-26 06:53:26.354215] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/B/Broker2_FTR_S_TE-XX_IN-A_51_HD_RIH_IOP_OV/Broker2_FTR-V1_S_TE-XX_IN-A_51_HD_20140526_RIH_IOP_OV/BROKER-2_R8_MPEG_AUDIO_250514.mxf:
> attempting to move from glustertest-disperse-29 to glustertest-disperse-16
>
>
>
>
> Subvolume0 to subvolume35
>
>
> root at glustertestdn001:# df -h
> Filesystem Size Used Avail Use% Mounted on Subvolume Details
> ========== ==== ==== ===== ==== =========== ==================
>
>
> /dev/sdb1 3.7T 3.6T 51G 99% /media/disk1 - glustertest-disperse-0
> /dev/sdc1 3.7T 3.3T 384G 90% /media/disk2 - glustertest-disperse-1
> /dev/sdd1 3.7T 3.6T 76G 98% /media/disk3 - glustertest-disperse-2
> /dev/sde1 3.7T 3.5T 214G 95% /media/disk4 - glustertest-disperse-3
> /dev/sdf1 3.7T 3.3T 365G 91% /media/disk5 - glustertest-disperse-4
> /dev/sdg1 3.7T 3.5T 173G 96% /media/disk6 - glustertest-disperse-5
> /dev/sdh1 3.7T 3.4T 284G 93% /media/disk7 - glustertest-disperse-6
> /dev/sdi1 3.7T 3.3T 373G 90% /media/disk8 - glustertest-disperse-7
> /dev/sdj1 3.7T 3.6T 122G 97% /media/disk9 - glustertest-disperse-8
> /dev/sdk1 3.7T 3.4T 247G 94% /media/disk10 - glustertest-disperse-9
> /dev/sdl1 3.7T 3.6T 108G 98% /media/disk11 - glustertest-disperse-10
> /dev/sdm1 3.7T 3.4T 252G 94% /media/disk12 - glustertest-disperse-11
> /dev/sdn1 3.7T 3.5T 241G 94% /media/disk13 - glustertest-disperse-12
> /dev/sdo1 3.7T 3.6T 136G 97% /media/disk14 - glustertest-disperse-13
> /dev/sdp1 3.7T 3.4T 247G 94% /media/disk15 - glustertest-disperse-14
> /dev/sdq1 3.7T 3.5T 170G 96% /media/disk16 - glustertest-disperse-15
> /dev/sdr1 3.7T 3.4T 257G 94% /media/disk17 - glustertest-disperse-16
> /dev/sds1 3.7T 3.4T 258G 94% /media/disk18 - glustertest-disperse-17
> /dev/sdt1 3.7T 3.5T 171G 96% /media/disk19 - glustertest-disperse-18
> /dev/sdu1 3.7T 3.5T 184G 96% /media/disk20 - glustertest-disperse-19
> /dev/sdv1 3.7T 3.4T 281G 93% /media/disk21 - glustertest-disperse-20
> /dev/sdw1 3.7T 3.3T 347G 91% /media/disk22 - glustertest-disperse-21
> /dev/sdx1 3.7T 3.5T 191G 95% /media/disk23 - glustertest-disperse-22
> /dev/sdy1 3.7T 3.5T 217G 95% /media/disk24 - glustertest-disperse-23
> /dev/sdz1 3.7T 3.4T 263G 93% /media/disk25 - glustertest-disperse-24
> /dev/sdaa1 3.7T 3.6T 66G 99% /media/disk26 - glustertest-disperse-25
> /dev/sdab1 3.7T 3.5T 225G 94% /media/disk27 - glustertest-disperse-26
> /dev/sdac1 3.7T 3.4T 310G 92% /media/disk28 - glustertest-disperse-27
> /dev/sdad1 3.7T 3.5T 159G 96% /media/disk29 - glustertest-disperse-28
> /dev/sdae1 3.7T 3.3T 373G 91% /media/disk30 - glustertest-disperse-29
> /dev/sdaf1 3.7T 3.4T 305G 92% /media/disk31 - glustertest-disperse-30
> /dev/sdag1 3.7T 3.5T 184G 96% /media/disk32 - glustertest-disperse-31
> /dev/sdah1 3.7T 3.5T 186G 96% /media/disk33 - glustertest-disperse-32
> /dev/sdai1 3.7T 3.3T 373G 91% /media/disk34 - glustertest-disperse-33
> /dev/sdaj1 3.7T 3.3T 361G 91% /media/disk35 - glustertest-disperse-34
> /dev/sdak1 3.7T 3.3T 375G 90% /media/disk36 - glustertest-disperse-35
>
>
>
>
>
>
> Subvolume36 to subvolume71
>
>
>
>
>
> root at glustertestdn011:/var/log/glusterfs# df -h
>
>
> Filesystem Size Used Avail Use% Mounted on SubVolume Detail
> ========== ==== ==== ===== ==== ========== ================
> /dev/sdc1 3.7T 475G 3.2T 13% /media/disk1 - glustertest-disperse-36
> /dev/sdd1 3.7T 467G 3.2T 13% /media/disk2 - glustertest-disperse-37
> /dev/sde1 3.7T 492G 3.2T 14% /media/disk3 - glustertest-disperse-38
> /dev/sdf1 3.7T 470G 3.2T 13% /media/disk4 - glustertest-disperse-39
> /dev/sdg1 3.7T 498G 3.2T 14% /media/disk5 - glustertest-disperse-40
> /dev/sdh1 3.7T 461G 3.2T 13% /media/disk6 - glustertest-disperse-41
> /dev/sdi1 3.7T 451G 3.2T 13% /media/disk7 - glustertest-disperse-42
> /dev/sdj1 3.7T 464G 3.2T 13% /media/disk8 - glustertest-disperse-43
> /dev/sdk1 3.7T 426G 3.3T 12% /media/disk9 - glustertest-disperse-44
> /dev/sdl1 3.7T 425G 3.3T 12% /media/disk10 - glustertest-disperse-45
> /dev/sdm1 3.7T 460G 3.2T 13% /media/disk11 - glustertest-disperse-46
> /dev/sdn1 3.7T 478G 3.2T 13% /media/disk12 - glustertest-disperse-47
> /dev/sdo1 3.7T 475G 3.2T 13% /media/disk13 - glustertest-disperse-48
> /dev/sdp1 3.7T 520G 3.2T 14% /media/disk14 - glustertest-disperse-49
> /dev/sdq1 3.7T 456G 3.2T 13% /media/disk15 - glustertest-disperse-50
> /dev/sdr1 3.7T 442G 3.3T 12% /media/disk16 - glustertest-disperse-51
> /dev/sds1 3.7T 494G 3.2T 14% /media/disk17 - glustertest-disperse-52
> /dev/sdt1 3.7T 495G 3.2T 14% /media/disk18 - glustertest-disperse-53
> /dev/sdu1 3.7T 426G 3.3T 12% /media/disk19 - glustertest-disperse-54
> /dev/sdv1 3.7T 471G 3.2T 13% /media/disk20 - glustertest-disperse-55
> /dev/sdw1 3.7T 531G 3.2T 15% /media/disk21 - glustertest-disperse-56
> /dev/sdx1 3.7T 576G 3.1T 16% /media/disk22 - glustertest-disperse-57
> /dev/sdy1 3.7T 411G 3.3T 12% /media/disk23 - glustertest-disperse-58
> /dev/sdz1 3.7T 455G 3.2T 13% /media/disk24 - glustertest-disperse-59
> /dev/sdaa1 3.7T 427G 3.3T 12% /media/disk25 - glustertest-disperse-60
> /dev/sdab1 3.7T 442G 3.3T 12% /media/disk26 - glustertest-disperse-61
> /dev/sdac1 3.7T 449G 3.2T 13% /media/disk27 - glustertest-disperse-62
> /dev/sdad1 3.7T 446G 3.3T 12% /media/disk28 - glustertest-disperse-63
> /dev/sdae1 3.7T 412G 3.3T 12% /media/disk29 - glustertest-disperse-64
> /dev/sdaf1 3.7T 480G 3.2T 13% /media/disk30 - glustertest-disperse-65
> /dev/sdag1 3.7T 485G 3.2T 14% /media/disk31 - glustertest-disperse-66
> /dev/sdah1 3.7T 453G 3.2T 13% /media/disk32 - glustertest-disperse-67
> /dev/sdai1 3.7T 410G 3.3T 12% /media/disk33 - glustertest-disperse-68
> /dev/sdaj1 3.7T 457G 3.2T 13% /media/disk34 - glustertest-disperse-69
> /dev/sdak1 3.7T 444G 3.3T 12% /media/disk35 - glustertest-disperse-70
> /dev/sdal1 3.7T 484G 3.2T 13% /media/disk36 - glustertest-disperse-71
>
>
>
>
> Is this the normal behavior of DHT in distributed disperse volume ?
>
> Requesting Pranith and Ashish to comment on this.
>
>
>
> Thanks
> Backer
>
>
>
>
>
>
>
> On Tue, Aug 25, 2015 at 12:36 PM, Susant Palai < spalai at redhat.com >
> wrote:
>
>
> Mohamed,
> Will investigate in to weighted rebalance behavior.
>
> Susant
>
> ----- Original Message -----
> From: "Mohamed Pakkeer" < mdfakkeer at gmail.com >
> To: "Susant Palai" < spalai at redhat.com >
> Cc: "Mathieu Chateau" < mathieu.chateau at lotp.fr >, "gluster-users" <
> gluster-users at gluster.org >, "Gluster Devel" < gluster-devel at gluster.org >
> Sent: Tuesday, 25 August, 2015 9:40:01 AM
> Subject: Re: [Gluster-users] cluster.min-free-disk is not working in
> distributed disperse volume
>
>
>
>
> Hi Sasant,
> We have created the disperse volume across nodes. We stopped all the
> upload operations and started the rebalance last night.After overnight
> re-balance, some harddisk is occupied 100% and some disks have 13% disk
> space.
>
>
> disk1 belongs to disperse-set-0 ..... disk36 belongs to disperse-set-35
>
>
> df -h result of one data node
>
>
>
> /dev/sdb1 3.7T 3.7T 545M 100% /media/disk1
> /dev/sdc1 3.7T 3.2T 496G 87% /media/disk2
> /dev/sdd1 3.7T 3.7T 30G 100% /media/disk3
> /dev/sde1 3.7T 3.5T 173G 96% /media/disk4
> /dev/sdf1 3.7T 3.2T 458G 88% /media/disk5
> /dev/sdg1 3.7T 3.5T 143G 97% /media/disk6
> /dev/sdh1 3.7T 3.5T 220G 95% /media/disk7
> /dev/sdi1 3.7T 3.3T 415G 89% /media/disk8
> /dev/sdj1 3.7T 3.6T 72G 99% /media/disk9
> /dev/sdk1 3.7T 3.5T 186G 96% /media/disk10
> /dev/sdl1 3.7T 3.6T 65G 99% /media/disk11
> /dev/sdm1 3.7T 3.5T 195G 95% /media/disk12
> /dev/sdn1 3.7T 3.5T 199G 95% /media/disk13
> /dev/sdo1 3.7T 3.6T 78G 98% /media/disk14
> /dev/sdp1 3.7T 3.5T 200G 95% /media/disk15
> /dev/sdq1 3.7T 3.6T 119G 97% /media/disk16
> /dev/sdr1 3.7T 3.5T 206G 95% /media/disk17
> /dev/sds1 3.7T 3.5T 193G 95% /media/disk18
> /dev/sdt1 3.7T 3.6T 131G 97% /media/disk19
> /dev/sdu1 3.7T 3.5T 141G 97% /media/disk20
> /dev/sdv1 3.7T 3.5T 243G 94% /media/disk21
> /dev/sdw1 3.7T 3.4T 299G 92% /media/disk22
> /dev/sdx1 3.7T 3.5T 163G 96% /media/disk23
> /dev/sdy1 3.7T 3.5T 168G 96% /media/disk24
> /dev/sdz1 3.7T 3.5T 219G 95% /media/disk25
> /dev/sdaa1 3.7T 3.7T 37G 100% /media/disk26
> /dev/sdab1 3.7T 3.5T 172G 96% /media/disk27
> /dev/sdac1 3.7T 3.4T 276G 93% /media/disk28
> /dev/sdad1 3.7T 3.6T 108G 98% /media/disk29
> /dev/sdae1 3.7T 3.3T 399G 90% /media/disk30
> /dev/sdaf1 3.7T 3.5T 240G 94% /media/disk31
> /dev/sdag1 3.7T 3.6T 122G 97% /media/disk32
> /dev/sdah1 3.7T 3.5T 147G 97% /media/disk33
> /dev/sdai1 3.7T 3.4T 342G 91% /media/disk34
> /dev/sdaj1 3.7T 3.4T 288G 93% /media/disk35
> /dev/sdak1 3.7T 3.4T 342G 91% /media/disk36
>
>
> disk1 belongs to disperse-set-0. Rebalancer logs shows, still rebalancer
> is trying to fill the disperse-set-0 after filling to 100%
>
>
>
>
> [2015-08-24 19:52:53.036622] E [MSGID: 109023]
> [dht-rebalance.c:672:__dht_check_free_space] 0-glustertest-dht: data
> movement attempted from node (glustertest-disperse-7) to node
> (glustertest-disperse-0) which does not have required free space for
> (/Packages/Features/MPEG/A/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_RIC_OV/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_20110521_RIC_OV/AMI-NEBO-C_R3_AUDIO_190511.mxf)
>
>
> [2015-08-24 19:52:53.042026] I [dht-rebalance.c:1002:dht_migrate_file]
> 0-glustertest-dht:
> /Packages/Features/MPEG/A/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_RIC_OV/AMEO-N-CHALLANGE_FTR_S_BEN-XX_IN-UA_51_HD_20110521_RIC_OV/AMINEBO-CHALLANGE_BEN_R1-2-3-4-5-6_MPEG_200511-reel-5-mpeg2.mxf:
> attempting to move from glustertest-disperse-13 to glustertest-disperse-0
>
>
> I think, cluster.weighted-rebalance and cluster.min-free-dis k have bugs
> for re-balancing the data, based on weight and disk free space.
>
>
>
>
> Thanks
> Backer
>
>
>
>
> On Mon, Aug 24, 2015 at 4:28 PM, Mohamed Pakkeer < mdfakkeer at gmail.com >
> wrote:
>
>
>
> Hi Susant,
>
>
> Thanks for your quick reply. We are not updating any files. Actually we
> are archiving video files on this cluster. I think there is a bug in
> cluster.min-free-disk.
>
>
> Also i would like to know about rebalance the cluster. Currently we have
> 20 nodes and 10 nodes hard disks are almost full . So we need to rebalance
> the data. If i run the rebalancer, it starts on first node(node1) and
> starts the migration process. The first node cpu usage is always high
> during rebalance compare with rest of the cluster nodes.To reduce the cpu
> usage of rebalancer datanode( node1), i peer a new node( without disk) for
> rebalance and start the rebalancer. It starts again the rebalancer on same
> node1. How can we run a rebalancer on a dedicated node?
>
>
> Also we are facing memory leaks in fixlayout and heal full operations.
>
>
> Regards
> Backer
>
>
>
>
> On Mon, Aug 24, 2015 at 2:57 PM, Susant Palai < spalai at redhat.com > wrote:
>
>
> Hi,
> Cluster.min-free-disk controls new file creation on the bricks. If you
> happen to write to the existing files on the brick and that is leading to
> brick getting full, then most probably you should run a rebalance.
>
> Regards,
> Susant
>
>
>
> ----- Original Message -----
> From: "Mathieu Chateau" < mathieu.chateau at lotp.fr >
> To: "Mohamed Pakkeer" < mdfakkeer at gmail.com >
> Cc: "gluster-users" < gluster-users at gluster.org >, "Gluster Devel" <
> gluster-devel at gluster.org >
> Sent: Monday, 24 August, 2015 2:47:00 PM
> Subject: Re: [Gluster-users] cluster.min-free-disk is not working in
> distributed disperse volume
>
>
>
>
> 720 brick! Respect !
> Le 24 août 2015 09:48, "Mohamed Pakkeer" < mdfakkeer at gmail.com > a écrit :
>
>
>
> Hi,
>
>
> I have a cluster of 720 bricks, all bricks are 4TB in size. I have change
> the cluster.min-free-disk default value 10% to 3%. So all the disks should
> have 3% minimum disk space free. But some cluster disks are getting full
> now. Is there any additional configuration for keeping some percentage of
> disk space kept free?
>
>
>
>
>
> Volume Name: glustertest
> Type: Distributed-Disperse
> Volume ID: 2b575b5c-df2e-449c-abb9-c56cec27e609
> Status: Started
> Number of Bricks: 72 x (8 + 2) = 720
> Transport-type: tcp
>
>
>
>
>
> Options Reconfigured:
> features.default-soft-limit: 95%
> cluster.min-free-disk: 3%
> performance.readdir-ahead: on
>
>
> df -h of one node
>
>
>
> /dev/sdb1 3.7T 3.6T 132G 97% /media/disk1
> /dev/sdc1 3.7T 3.2T 479G 88% /media/disk2
> /dev/sdd1 3.7T 3.6T 109G 98% /media/disk3
>
>
> Any help will be greatly appreciated.
>
>
>
>
> Regards
> Backer
>
>
>
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150828/ac2006a1/attachment.html>
More information about the Gluster-users
mailing list