[Gluster-users] Giving up [ was: Re: read-subvolume]

[Jeff Darcy]

On 07/10/2013 07:01 AM, Allan Latham wrote:
> I have a simple scenario and it just simply doesn't work. Reading over
> the network when the file is available locally is plainly wrong. Our
> application cannot take the performance hit nor the extra network traffic.

Another victim of our release process.  :(  Code was added to choose the local 
subvolume whenever possible in *June 2012* (commit 0baa65b6).  Further fixes 
and related changes, including a user-submitted patch to force this choice for 
sites with more complex needs, have gone in since then.  None of them have made 
it into a release yet, since 3.4 is still in beta and the changes have not been 
backported into 3.3.anything (including 3.3.1 which I see you were using).  All 
I can offer is an apology.

> 1. get a simple minimalist configuration working - 2 hosts and
> replication only.
> 2. make it bomb-proof.
> 2a. it must cope with network failures, random reboots etc.
> 2b. if it stops it has to auto-recover quickly.
> 2c. if it can't it needs thorough documentation and adequate logs so a
> reasonable sysop can rescue it.

This is one of my own pet peeves.  I will personally be working on the 
internals documentation soon, so users will at least have a chance of 
understanding what the often-cryptic log messages really mean.  Improvements to 
logging, event reporting, and so on are also ongoing, albeit slowly and not 
under my direct purview.

> 2d. it needs a fast validation scanner which verifies that data is where
> it should be and is identical everywhere (md5sum).

How fast is fast?  What would be an acceptable time for such a scan on a volume 
containing (let's say) ten million files?

> 3. make it efficient (read local whenever possible - use rsync
> techniques - remove scalability obstacles so it doesn't get
> exponentially slower as more files are replicated)

Can you explain "exponentially"?  The time for a full scan should increase 
*linearly* with number of files.  That's bad enough, and it's why we're 
starting to get away from reliance on full scans in favor of logging or 
journaling approaches, but if you're seeing exponential behavior then something 
is amiss.

> 4. when that works expand to multiple hosts and clever distribution
> techniques.

That would be a fine sentiment for a new project, but it's not really an option 
when there are already thousands of users relying on the "clever distribution 
techniques" and many other features in production.  We do have to fix their 
bugs too, so we can't devote all of our resources to improving or 
reimplementing replication.  Believe me, I wish we could.

Thank you for your constructive feedback.  I hope that we can use it to make 
things better for everyone.