[Gluster-devel] Re; Load balancing ...
Mickey Mazarick
mic at digitaltadpole.com
Wed Apr 30 15:58:46 UTC 2008
We have a work around for this so that clients don't see a "blocked"
situation. we have a 6 node cluster and whenever a server node comes up
it lists all the files on it's "slice" of the gluster mount and then
does a find on those files on the main mounted volume. In other words
each server is also a client, and does the heal process right after
starting up.
-Mic
Gareth Bult wrote:
> Ok, I'm afraid I'm not agreeing with some of the things you're saying, but that doesn't really get us anywhere.
>
> I think the bottom line is, having read or write operations block for any length of time in order to merge a node back into the cluster following an outage simply isn't "real world". "any length of time" will vary for different operators, for me it's seconds, for anyone running real-time systems or VoIP for example, it might be milliseconds.
>
> My feeling is that self-heal "on open" is "nice", easy to code and maintain, neat, and I can see "how it came to be".
>
> It's now however practical for a commercial solution, healing needs to be a background process, not a foreground process.
>
> The only question I really have is, what sort of timescale are we looking at before GlusterFS has the kind of capabilities (in context) that one would expect of a production clustering filesystem ?
> (I think we all know hash/sync would be an interim solution)
>
> (I know it's "different", but just as a point of reference, DRBD syncs in the background, as does Linux software raid .. can you imagine anyone using either if they tried to make people wait while they did a foreground sync ??)
>
> Gareth.
>
>
> ----- Original Message -----
> From: gordan at bobich.net
> To: gluster-devel at nongnu.org
> Sent: Wednesday, April 30, 2008 1:56:12 PM GMT +00:00 GMT Britain, Ireland, Portugal
> Subject: Re: [Gluster-devel] Re; Load balancing ...
>
>
>
> On Wed, 30 Apr 2008, Gareth Bult wrote:
>
>
>>> It would certainly ber beneficial in the cases when the network speed
>>> is slow (e.g. WAN replication).
>>>
>> So long as it's server side AFR and not client-side ... ?
>>
>
> Sure.
>
>
>> I'm guessing there would need to be some server side logic to ensure
>> that local servers generated their own hashes and only exchanged the
>> hashes over the network rather than the data ?
>>
>
> Indeed - same as rsync does.
>
>
>>> Journal per se wouldn't work, because that implies fixed size and write-ahead logging.
>>> What would be required here is more like the snapshot style undo logging.
>>>
>> A journal wouldn't work ?!
>> You mean it's effectiveness would be governed by it's size?
>>
>
> Among other things. A "journal" just isn't suitable for this sort of
> thing.
>
>
>>> 1) Categorically establish whether each server is connected and up to date
>>> for the file being checked, and only log if the server has disconnected.
>>> This involves overhead.
>>>
>> Surely you would log anyway, as there could easily be latency between an
>> actual "down" and one's ability to detect it .. in which case detecting
>> whether a server has disconnected it a moot point.
>>
>
> Not really. A connected client/server will have a live/working TCP
> connection open. Read-locks don't matter as they can be served locally,
> but when a write occurs, the file gets locked. If a remote machine doesn't
> ack the lock, and/or it's TCP connection resets, then it's safe to assume
> that it's not connected.
>
>
>> In terms of the
>> overhead of logging, I guess this would be a decision for the sysadmin
>> concerned, whether the overhead of logging to a journal was worthwhile
>> .vs. the potential issues involved in recovering from an outage?
>>
>
> That complicates things further, then. You'd essentially have asynchronous
> logging/replication. At that point you pretty much have to log all writes
> all the time. That means potentially huge space and speed overheads.
>
>
>> From my point of view, if journaling halved my write performance (which
>> it wouldn't) I wouldn't even have to think about it.
>>
>
> Actually, saving an undo-log a-la snapshots, which is what would be
> required, _WOULD_ halve your write performance on all surviving servers if
> one server was out. If multiple servers were out, you could probably work
> around some of this with merging/splitting the undo logs for various
> machines, so your write performance would generally be around 1/2 of
> standard, but wouldn't end up degrading to 1/n+1 where n is the number of
> failed servers for which the logging needs to be done.
>
>
>>> The problem that arises then is that the fast(er) resyncs on small changes
>>> come at the cost of massive slowdown in operation when you have multiple
>>> downed servers. As the number of servers grows, this rapidly stops being a
>>> workable solution.
>>>
>> Ok, I don't know about anyone else, but my setups all rely on
>> consistency rather than peaks and troughs. I'd far rather run a journal
>> at half potential speed, and have everything run at that speed all the
>> time .. than occasionally have to stop the entire setup while the system
>> recovers, or essentially wait for 5-10 minutes while the system re-syncs
>> after a node is reloaded.
>>
>
> There may be a way to address the issue of halting the rest of the cluster
> during the sync, though. Read lock on a syncing file shouldn't stop other
> read locks. Of course, it will block writes while the file syncs and the
> reading app finishes the operation.
>
> Gordan
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
--
More information about the Gluster-devel
mailing list