[Gluster-devel] Local vs unify

[gordan at bobich.net]

On Mon, 28 Apr 2008, Paul Arch wrote:

>>> Thanks for supporting our design. We are working towards fixing those few
> glitches!
>>> But true that things are not changing as fast as we have wished. Each new
> idea needs
>>> time to get converted into code, get tested. Hence bit delay in things.
>>
>> No problem and thank you for this email, it has answered a major issue
>> for me .. I am of course going to ask;
>> a. Any timescale on the metadata changes?
>> b. How much of a difference will it make.. will we be approaching
>> local(ish) speeds .. or are we just talking x2 of current?
>
>> I imagine that would depend on the metadata expiry timeouts. If it's set
>> to 100ms, the chances are that you won't see much improvement. If it's set
>> for 100 seconds, it'll go as fast as local FS for cached data but you'll
>> be working on FS state that might as well be imaginary in some cases. No
>> doubt someone will then complain about the fact that posix semantics no
>> longer work.
>
> <snip>
>
> I have been following this thread and the metadata stuff does interest me -
> we have millions and millions of small files.
>
> In the above situation though, I would of thought knowing all of the inputs
> into the system ( ie - gluster knows that state everything is in, as long as
> no-one enters and changes things from outside of the mechanism in the
> back-ground ) could see some fair potential for caching the meta data.  If
> the system is in a degraded state sure you wouldn't and shouldn't trust this
> cache, but all things being equal and happy, why can't we trust a good sized
> cache metadata is AFR/unity/whatever is reporting the system is happy and
> operational ?

This relates to the point I made a few days ago on the other thread. You 
_could_ do this, but in order to do that, you'd have to change the 
sync-on-read paradigm and couple the systems much more tightly. This would 
likely involve things like mandatory fencing requirements which are 
currently avoided.

If you have a read-lock on a file, you cannot get a write-lock on it, so 
you could potentially sacrifice write-lock performance for read-locking in 
that case, by making read-locks always available without external checking 
against other nodes unless a write lock is in place (which needs to be 
broadcast and acknowledged by _all_ nodes in the cluster).

This is also made more difficult with unify or striping because the data 
is remote in the first place, so you have to retrieve the metadata at 
least from the server - unless you want to cache it locally, which would 
gain break posix semantics.

Note - NFS is not posix. You can set metadata cache expiry on NFS. NFS 
also has the advantage that the data is on _one_ server, so even if 
there was some form of locking that reliably works over NFS available 
(there isn't, but for the sake of the argument, if there was) there would 
still be no concept of chasing locks across the cluster to make sure the 
mirrors are consistent before granting them.

In short - comparing NFS to GlusterFS isn't really meaningful.

Gordan