[Gluster-users] cluster.min-free-disk separate for each, brick

Dan Bretherton d.a.bretherton at reading.ac.uk
Sat Nov 19 14:12:22 UTC 2011

On 29/09/11 12:28, Dan Bretherton wrote:
> On 08/09/11 23:51, Dan Bretherton wrote:
>>> On Wed, Sep 7, 2011 at 4:27 PM, Dan Bretherton 
>>> <d.a.bretherton at reading.ac.uk <mailto:d.a.bretherton at reading.ac.uk>> 
>>> wrote:
>>>     On 17/08/11 16:19, Dan Bretherton wrote:
>>>             Dan Bretherton wrote:
>>>                 On 15/08/11 20:00, gluster-users-request at gluster.org
>>>                 <mailto:gluster-users-request at gluster.org> wrote:
>>>                     Message: 1
>>>                     Date: Sun, 14 Aug 2011 23:24:46 +0300
>>>                     From: "Deyan Chepishev -
>>>                     SuperHosting.BG"<dchepishev at superhosting.bg
>>>                     <mailto:dchepishev at superhosting.bg>>
>>>                     Subject: [Gluster-users] cluster.min-free-disk
>>>                      separate for each
>>>                        brick
>>>                     To: gluster-users at gluster.org
>>>                     <mailto:gluster-users at gluster.org>
>>>                     Message-ID:<4E482F0E.3030604 at superhosting.bg
>>>                     <mailto:4E482F0E.3030604 at superhosting.bg>>
>>>                     Content-Type: text/plain; charset=UTF-8;
>>>                     format=flowed
>>>                     Hello,
>>>                     I have a gluster set up with very different
>>>                     brick sizes.
>>>                     brick1: 9T
>>>                     brick2: 9T
>>>                     brick3: 37T
>>>                     with this configuration if I set the parameter
>>>                     cluster.min-free-disk to 10% it
>>>                     applies to all bricks which is quite
>>>                     uncomfortable with these brick sizes,
>>>                     because 10% for the small bricks are ~ 1T but
>>>                     for the big brick it is ~3.7T and
>>>                     what happens at the end is that if all brick go
>>>                     to 90% usage and I continue
>>>                     writing, the small ones eventually fill up to
>>>                     100% while the big one has enough
>>>                     free space.
>>>                     My question is, is there a way to set
>>>                     cluster.min-free-disk per brick instead
>>>                     setting it for the entire volume or any other
>>>                     way to work around this problem ?
>>>                     Thank you in advance
>>>                     Regards,
>>>                     Deyan
>>>                 Hello Deyan,
>>>                 I have exactly the same problem and I have asked
>>>                 about it before - see links below.
>>>                 http://community.gluster.org/q/in-version-3-1-4-how-can-i-set-the-minimum-amount-of-free-disk-space-on-the-bricks/
>>>                 http://gluster.org/pipermail/gluster-users/2011-May/007788.html
>>>                 My understanding is that the patch referred to in
>>>                 Amar's reply in the May thread prevents a
>>>                 "migrate-data" rebalance operation failing by
>>>                 running out of space on smaller bricks, but that
>>>                 doesn't solve the problem we are having.  Being able
>>>                 to set min-free-disk for each brick separately would
>>>                 be useful, as would being able to set this value as
>>>                 a number of bytes rather than a percentage.
>>>                  However, even if these features were present we
>>>                 would still have a problem when the amount of free
>>>                 space becomes less than min-free-disk, because this
>>>                 just results in a warning message in the logs and
>>>                 doesn't actually prevent more files from being
>>>                 written.  In other words, min-free-disk is a soft
>>>                 limit rather than a hard limit.  When a volume is
>>>                 more than 90% full there may still be hundreds of
>>>                 gigabytes of free space spread over the large
>>>                 bricks, but the small bricks may each only have a
>>>                 few gigabytes left of even less.  Users do "df" and
>>>                 see lots of free space in the volume so they
>>>                 continue writing files.  However, when GlusterFS
>>>                 chooses to write a file to a small brick, the write
>>>                 fails with "device full" errors if the file grows
>>>                 too large, which is often the case here with files
>>>                 typically several gigabytes in size for some
>>>                 applications.
>>>                 I would really like to know if there is a way to
>>>                 make min-free-disk a hard limit.  Ideally, GlusterFS
>>>                 would chose a brick on which to write a file based
>>>                 on how much free space it has left rather than
>>>                 choosing a brick at random (or however it is done
>>>                 now).  That would solve the problem of non-uniform
>>>                 brick sizes without the need for a hard
>>>                 min-free-disk limit.
>>>                 Amar's comment in the May thread about QA testing
>>>                 being done only on volumes with uniform brick sizes
>>>                 prompted me to start standardising on a uniform
>>>                 brick size for each volume in my cluster.  My
>>>                 impression is that implementing the features needed
>>>                 for users with non-uniform brick sizes is not a
>>>                 priority for Gluster, and that users are all
>>>                 expected to use uniform brick sizes.  I really think
>>>                 this fact should be stated clearly in the GlusterFS
>>>                 documentation, in the sections on creating volumes
>>>                 in the Administration Guide for example.  That would
>>>                 stop other users from going down the path that I did
>>>                 initially, which has given me a real headache
>>>                 because I am now having to move tens of terabytes of
>>>                 data off bricks that are larger than the new
>>>                 standard size.
>>>                 Regards
>>>                 Dan.
>>>             Hello,
>>>             This is really bad news, because I already migrated my
>>>             data and I just realized that I am screwed because
>>>             Gluster just does not care about the brick sizes.
>>>             It is impossible to move to uniform brick sizes.
>>>             Currently we use 2TB  HDDs, but the disks are growing
>>>             and soon we will probably use 3TB hdds or whatever other
>>>             larges sizes appear on the market. So if we choose to
>>>             use raid5 and some level of redundancy (for example
>>>             6hdds in raid5, no matter what their size is) this
>>>             sooner or later will lead us to non uniform bricks which
>>>             is a problem and it is not correct to expect that we
>>>             always can or want to provide uniform size bricks.
>>>             With this way of thinking if we currently have 10T from
>>>             6x2T in hdd5, at some point when there is a 10T on a
>>>             single disk we will have to use no raid just because
>>>             gluster can not handle non uniform bricks.
>>>             Regards,
>>>             Deyan
>>>         I think Amar might have provided the answer in his posting
>>>         to the thread yesterday, which has just appeared in my
>>>         autospam folder.
>>>         http://gluster.org/pipermail/gluster-users/2011-August/008579.html
>>>             With size option, you can have a hardbound on min-free-disk
>>>         This means that you can set a hard limit on min-free-disk,
>>>         and set a value in GB that is bigger than the biggest file
>>>         that is ever likely to be written.  This looks likely to
>>>         solve our problem and make non-uniform brick sizes a
>>>         practical proposition.  I wish I had known about this back
>>>         in May when I embarked on my cluster restructuring exercise;
>>>         the issue was discussed in this thread in May as well:
>>>         http://gluster.org/pipermail/gluster-users/2011-May/007794.html
>>>         Once I have moved all the data off the large bricks and
>>>         standardised on a uniform brick size, it will be relatively
>>>         easy to stick to this because I use LVM.  I create logical
>>>         volumes for new bricks when a volume needs extending.  The
>>>         only problem with this approach is what happens when the
>>>         amount of free space left on a server is less than the size
>>>         of the brick you want to create.  The only option then would
>>>         be to use new servers, potentially wasting several TB of
>>>         free space on existing servers.  The standard brick size for
>>>         most of my volumes is 3TB, which allows me to use a mixture
>>>         of small servers and large servers in a volume and limits
>>>         the amount of free space that would be wasted if there
>>>         wasn't quite enough free space on a server to create another
>>>         brick.  Another consequence of having 3TB bricks is that a
>>>         single server typically has two more more bricks belonging
>>>         to a the same volume, although I do my best to distribute
>>>         the volumes across different servers in order to spread the
>>>         load.  I am not aware of any problems associated with
>>>         exporting multiple bricks from a single server and it has
>>>         not caused me any problems so far that I am aware of.
>>>         -Dan.
>>>     Hello Deyan,
>>>     Have you tried giving min-free-disk a value in gigabytes, and if
>>>     so does it prevent new files being written to your bricks when
>>>     they are nearly full?  I recently tried it myself and found that
>>>     min-free-disk had no effect all.  I deliberately filled my
>>>     test/backup volume and most of the bricks became 100 full.  I
>>>     set min-free-disk to "20GB", as reported in "gluster volume ...
>>>     info" below.
>>>     cluster.min-free-disk: 20GB
>>>     Unless I am doing something wrong it seems as though we can not
>>>     "have a hardbound on min-free-disk" after all, and uniform brick
>>>     size is therefore an essential requirement.  It still doesn't
>>>     say that in the documentation, at least not in the volume
>>>     creation sections.
>>>     -Dan.
>>> On 08/09/11 06:35, Raghavendra Bhat wrote:
>>> > This is how it is supposed to work.
>>> >
>>> > Suppose a distribute volume is created with 2 bricks. 1st brick is 
>>> having 25GB of free space, 2nd disk has 35 GB of free space. If one 
>>> sets a 30GB of minimum-free-disk through volume set (gluster volume 
>>> set <volname> min-free-disk 30GB), then whenever files are created, 
>>> if the file is hashed to the 1st brick (which has 25GB of free 
>>> space), then actual file will be created in the 2nd brick to which a 
>>> linkfile will be created in the 1st brick. So the linkfile points to 
>>> the actual file. A warning message indicating minimum free disk 
>>> limit has been crosses and adding more nodes will be printed in the 
>>> glusterfs log file. So any file which is hashed to the 1st brick 
>>> will be created in the 2nd brick.
>>> >
>>> > Once the free space of 2nd brick also comes below 30 GB, then the 
>>> files will be created in the respective hashed bricks only. There 
>>> will be a warning message in the log file about the 2nd brick also 
>>> crossing the minimum free disk limit.
>>> >
>>> > Regards,
>>> > Raghavendra Bhat
>> Dear Raghavendra,
>> Thanks for explaining this to me.  This mechanism should allow a 
>> volume to function correctly with non-uniform brick sizes even though 
>> min-free-disk is not a hard limit.  I can understand now why I had so 
>> many problems with the default value of 10% for min-free-disk.  10% 
>> of a large brick can be very large compared to 10% of a small brick, 
>> so when they started filling up at the same rate after all had less 
>> than 10% free space the small bricks usually filled up long before 
>> large ones, giving "device full" errors even when df still showed a 
>> lot of free space in the volume.  At least now we can minimise this 
>> effect by setting min-free-disk to a value in GB.
>> -Dan.
> Dear Raghavendra,
> Unfortunately I am still having problems with some bricks filling up 
> completely, despite having "cluster.min-free-disk: 20GB".  In one case 
> I am still seeing warnings about bricks being nearly full in 
> percentage terms in the client logs, so I am wondering if the volume 
> is still using cluster.min-free-disk: 10%, and ignoring the 20GB 
> setting I changed it to.  When I changed cluster.min-free-disk should 
> this have taken effect immediately is there something else I should 
> have done to activate the change?
> In your example above, suppose there are 9 bricks instead of 2 bricks 
> (as in my volume), and they all have less than 30GB free space except 
> for one which is nearly empty, is GlusterFS clever enough to find that 
> nearly empty brick every time when creating new files?  I expected all 
> new files to be created in my nearly empty brick but that has not 
> happened.  Some files have gone in there but most have gone to nearly 
> full bricks, one of which has now filled up completely.  I have done 
> rebalance...fix-layout a number of times.  What can I do to fix this 
> problem?  The volumes with one or more full bricks are unusable 
> because users are getting "device full" errors for some writes even 
> though both volumes are showing several TB free space.
> Regards
> -Dan Bretherton.

Dear All,
If anyone is interested, I managed to produce the expected behaviour by 
setting min-free-disk to 300GB rather than 30GB.  300GB is is 
approximately 10% of the size of most of the bricks in the volume.  I 
don't understand why setting min-free-disk to 30GB (about 1% of the 
brick) didn't work; maybe it is too close to the limit for some reason.  
I wonder if the default value of min-free-disk=10% is significant.  It 
seems that for non-uniform brick sizes, the correct approach is to set 
min-free-disk to a value in GB that is approximately 10% of the brick 
size in each case.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111119/54159810/attachment.html>

More information about the Gluster-users mailing list