[Gluster-users] How to correctly distribute OpenStack VM files...

[Joel Young]

Are you actually observing this?  With cryptographic hashes being
effectively uniform the probability of such an extreme distribution is
extraordinarily low.
On Aug 2, 2013 6:23 AM, "Xavier Trilla" <xavier.trilla at silicontower.net>
wrote:

>  Hi,****
>
> ** **
>
> We have been playing for a while with GlusterFS (Now with ver 3.4). We are
> running tests and playing with it to check if GlusterFS  can be really used
> as the distributed storage for OpenStack block storage (Cinder) as new
> features in KVM, GlusterFS and OpenStack are pointing to GlusterFS as the
> future of OpenStack open source block and object storage. ****
>
> ** **
>
> But we’ve found a problem just when we started playing with GlusterFS… The
> way distribute translator (DHT) balances the load. I mean, we understand
> and see the benefits of metadata less setup. Using hashes based on
> filenames and assigning a hash range to each brick is clever, reliable and
> fast, but from our understanding there is a big problem when it comes to
> storing VM images of a OpenStack deployment. ****
>
> ** **
>
> I mean, OpenStack Block Storage (Cinder) assigns a name to each volume it
> creates (GUID), so GlusterFS does a hash of the filename and decides in
> which brick it should be stored. But as in this scenario we don’t have many
> files (I mean, we would just have one big file per VM) we may end with a
> really unbalanced storage. ****
>
> ** **
>
> Let’s say we have a 4 bricks setup with DHT distribute, and we want to
> store 100 VMs there, so the ideal scenario would be:****
>
> ** **
>
> Brick1: 25 VMs****
>
> Brick2: 25 VMs****
>
> Brick3: 25 VMs****
>
> Brick4: 25 VMs****
>
> ** **
>
> As VMs are IO intensive it’s really important to correctly balance the
> load, as each brick has a limited amount of IOPS, but as DHT is just based
> on a filename HASH, we could end with something like the following scenario
> (Or even worse): ****
>
> ** **
>
> Brick1: 50 VMs****
>
> Brick2: 10 VMs****
>
> Brick3: 35 VMs****
>
> Brick4: 5 VMs****
>
> ** **
>
> And if we scale this out, things may get even worse. I mean, we may end
> with almost all VM file in one or two bricks and all the other bricks
> almost empty. And if we use growing VM disk image files like qcow2 the
> option “min-free-disk” will not prevent all VMs disk image files being
> stored in the same brick. So, I understand DHT works well for large amount
> of small files, but for few big IO intensive files doesn’t seem to be a
> really good solution… (I mean, we are looking for a solution able to handle
> around 32 bricks and around 1500 VM for the initial deployment and able to
> scale up to 256 bricks and 12000 VMs :/ )****
>
> ** **
>
> So, anybody has a suggestion about how to handle this? I mean so far we
> only see two options: Either using legacy unify translator with ALU
> scheduler or either use cluster/stripe translator with a big block-size so
> at least load gets balanced across all bricks in some way.  But obviously
> we don’t like unify as it needs a namespace brick, and using stripping
> seems to have an impact on performance and really complicates
> backup/restore/recovery strategies. ****
>
> ** **
>
> So, suggestions? :) ****
>
> ** **
>
> Thanks!****
>
> ** **
>
> Saludos cordiales,****
>
> Xavier Trilla P.****
>
> Silicon Hosting <https://siliconhosting.com/>****
>
> ** **
>
> ¿Sabías que ahora en SiliconHosting ****
>
> resolvemos tus dudas técnicas gratis?****
>
> ** **
>
> Más información en: siliconhosting.com/qa/****
>
> ** **
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130802/a54063c6/attachment.html>