[Gluster-users] how well will this work

[Miles Fidelman]

Joe,

Thanks for the details, but I'm having just a bit of a problem parsing 
and picturing the description.  Any possibility of walking this through 
from the bottom up - hardware (server + n drives) -> drive partitioning 
-> lvm setup -> gluster config -> VM setup?

Thanks again,

Miles

Joe Julian wrote:
> I have 3 servers with replica 3 volumes, 4 bricks per server on lvm 
> partitions that are placed on each of 4 hard drives, 15 volumes 
> resulting in 60 bricks per server. One of my servers is also a kvm 
> host running (only) 24 vms.
>
> Each vm image is only 6 gig, enough for the operating system and 
> applications and is hosted on one volume. The data for each 
> application is hosted on its own GlusterFS volume.
>
> For mysql, I set up my innodb store to use 4 files (I don't do 1 file 
> per table), each file distributes to each of the 4 replica subvolumes. 
> This balances the load pretty nicely.
>
> I don't really do anything special for anything else, other than the 
> php app recommendations I make on my blog (http://joejulian.name) 
> which all have nothing to do with the actual filesystem.
>
> The thing that I think some people (even John Mark) miss apply is that 
> this is just a tool. You have to engineer a solution using the tools 
> you have available. If you feel the positives that GlusterFS provides 
> outweigh the negatives, then you will simply have to engineer a 
> solution that suits your end goal using this tool. It's not a question 
> of whether it works, it's whether you can make it work for your use case.
>
> On 12/27/2012 03:00 PM, Miles Fidelman wrote:
>> Ok... now that's diametrically the opposite response from Dan Cyr's 
>> of a few minutes ago.
>>
>> Can you say just a bit more about your configuration - how many 
>> nodes, do you have storage and processing combined or separated, how 
>> do you have your drives partitioned, and so forth?
>>
>> Thanks,
>>
>> Miles
>>
>>
>> Joe Julian wrote:
>>> Trying to return to the actual question, the way I handle those is 
>>> to mount gluster volumes that host the data for those tasks from 
>>> within the vm. I've done that successfully since 2.0 with all of 
>>> those services.
>>>
>>> The limitations that others are expressing have as much to do with 
>>> limitations placed on their own designs as with their hardware. 
>>> Sure, there are other less stable and/or scalable systems that are 
>>> faster, but with proper engineering you should be able to build a 
>>> system that meets those design requirements.
>>>
>>> The one piece that wasn't there before but now is in 3.3 is the 
>>> "locking and performance problems during disk rebuilds" which is now 
>>> done at a much more granular level and I have successfully 
>>> self-healed several vm images simultaneously while doing it on all 
>>> of them without any measurable delays.
>>>
>>> Miles Fidelman <mfidelman at meetinghouse.net> wrote:
>>>
>>>     Joe Julian wrote:
>>>
>>>         It would probably be better to ask this with end-goal
>>>         questions instead of with a unspecified "critical feature"
>>>         list and "performance problems".
>>>
>>>
>>>     Ok... I'm running a 2-node cluster that's essentially a mini 
>>> cloud stack
>>>     - with storage and processing combined on the same boxes. I'm 
>>> running a
>>>     production VM that hosts a mail server, list server, web server, 
>>> and
>>>     database; another production VM providing a backup server for the
>>>     cluster and for a bunch of desktop machines; and several VMs 
>>> used for a
>>>     variety of development and testing purposes. It's all backed by a
>>>     storage stack consisting of linux raid10 -> lvm -> drbd, and uses
>>>     pacemaker for high-availability failover of the
>>>     production VMs.  It all
>>>     performs reasonably well under moderate load (mail flows, web 
>>> servers
>>>     respond, database transactions complete, without notable user-level
>>>     delays; queues don't back up; cpu and io loads stay within 
>>> reasonable
>>>     bounds).
>>>
>>>     The goals are to:
>>>     - add storage and processing capacity by adding two more nodes - 
>>> each
>>>     consisting of several CPU cores and 4 disks each
>>>     - maintain the flexibility to create/delete/migrate/failover 
>>> virtual
>>>     machines - across 4 nodes instead of 2
>>>     - avoid having to play games with pairwise DRBD configurations 
>>> by moving
>>>     to a clustered filesystem
>>>     - in essence, I'm looking to do what Sheepdog purports to do, 
>>> except in
>>>     a Xen environment
>>>
>>>     Earlier versions of gluster had reported problems with:
>>>     - supporting databases
>>>     - supporting VMs
>>>     - locking and performance problems during disk rebuilds
>>>     - and... most of the gluster documentation implies that it's
>>>     preferable
>>>     to separate storage nodes from processing nodes
>>>
>>>     It looks like Gluster 3.2 and 3.3 have addressed some of these 
>>> issues,
>>>     and I'm trying to get a general read on whether it's worth 
>>> putting in
>>>     the effort of moving forward with some experimentation, or 
>>> whether this
>>>     is a non-starter.  Is there anyone out there who's tried to run 
>>> this
>>>     kind of mini-cloud with gluster?  What kind of results have you 
>>> had?
>>>
>>>
>>>
>>>         On 12/26/2012 08:24 PM, Miles Fidelman wrote:
>>>
>>>             Hi Folks, I find myself trying to expand a 2-node
>>>             high-availability cluster from to a 4-node cluster. I'm
>>>             running Xen virtualization, and currently using DRBD to
>>>             mirror data, and pacemaker to failover cleanly. The thing
>>>             is, I'm trying to add 2 nodes to the cluster, and DRBD
>>>             doesn't scale. Also, as a function of rackspace limits,
>>>             and the hardware at hand, I can't separate storage nodes
>>>             from compute nodes - instead, I have to live with 4 nodes,
>>>             each with 4 large drives (but also w/ 4 gigE ports per
>>>             server). The obvious thought is to use Gluster to assemble
>>>             all the drives into one large storage pool, with
>>>             replication. But.. last time I looked at this (6 months or
>>>             so back), it looked like some of the critical features
>>>             were brand new, and performance seemed to be a problem in
>>>             the configuration I'm thinking of. Which leads me to my
>>>             question: Has the situation improved to the point that I
>>>             can use Gluster this way? Thanks very much, Miles Fidelman
>>>
>>> ------------------------------------------------------------------------ 
>>>
>>>         Gluster-users mailing list Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>
>>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users


-- 
In theory, there is no difference between theory and practice.
In practice, there is.   .... Yogi Berra