[Gluster-devel] Re; Load balancing ...

[gordan at bobich.net]

On Thu, 1 May 2008, Martin Fick wrote:

>> Sounds like a good idea. The next question is where
>> to keep the log. 1 log per file? 1 log per
>> directory?
>> How to store them? Shadow files? Separate
>> shadow volume? A shadow volume might be a good idea
>> because it keeps the  main source mounted directory
>> exactly the same as a normal directory.
>
> I would start as simple as possible and adapt as
> necessary if you run into a performance problem.  The
> simplest design would probably be a shadow volume with
> one log per file with the a sparse mirrored directory
> structure.

Indeed, that's exactly what I was thinking. You would effectively need a 
container, like a namespace, to unify the two.

> Logs could be 24(?) bytes concatenated one
> after another making appending easy and reliable.  Or
> at a minor space cost (but potential added
> portability/extendability), each log file could even
> be a colon delimited line based ascii file (please
> don't anyone suggest an xml file!)
>
>  version1:start2:span2
>  version2:start2:span2
>  ...

If it's fixed length pointers (or in fact fixed length records), I'd go 
with packed binary format for efficiency and speed. These will have to be 
written to on every write. There would also need to be a header that 
states where the roll-over point is. Effectively, the log would be an RRD.

> Having a separate log file for each real file also
> makes it easy to code up some optimizations, for
> example: it would be easy to lookup the size of the
> log and the size of the real file.  As soon as the log
> becomes bigger than the real file it is no longer
> worth keeping as is!  It also makes it real easy to
> just delete the log if the real file is deleted.

Maybe have the default log be about 0.5% of the file in powers of 2, and 
not used for files below a certain size. Maybe grow/shrink it when it 
would exceed one step in powers of 2 from it's intended 0.5% size. This 
would mean that as the file grows, the log increases, but the log 
extention gets exponentially more rare. log truncation could be left until 
a suitable roll-over point. If you are syncing inodes, then that is 
typically 4KB, and a log entry would be, as you said, 24 bytes. That makes 
a log entry for a changed inode block about 1/170, which is about right 
for the 0.5% ball park.

> Another nice optimizer could make intelligent
> decisions about which log files to delete when the
> shadow volume starts to fill up.  By simply examining
> the size of each log versus the size of the real file
> one can set an upper bounds on how much transfer data
> the log could be saving (a real estimate would require
> adding all the spans together in the log file taking
> into account overlapping sections).

Sure - if you want to keep volumes separate. Or you could just maue sure 
that your log volume is always at least 1/170 of the data volume it's 
shadowing. Possibly a bit more for a safety margin with the lazy log 
resizing - around 1% ought to suffice for most sane cases.

> Finally, it would
> allow an admin to prune the shadow volume manually of
> whichever logs he chooses to prune.  An ascii file
> would make it easy to script various pruners.

I think that starts getting potentially dangerous. I think just having the 
logs volume at about 1% of the data volume would be better. Of course, if 
you keep both on the same physical volume, it won't matter.

> It would be nice to design the shadow volume so that
> it can be removed from the picture at any time without
> corrupting anything.

You already covered that with the sparse shadow volume tree. If there's no 
log, you resync the whole file.

> It would also be nice to ensure
> that the journal translator can handle an out of space
> condition.  This way each server is not required to
> even have the same size journal volume if any at all.

Note that this gets into the chicken and the egg problem - the log files 
would still need to be syncable directly using the current method - or 
you'd need a journal for your journalling volume. But if the journal is 
typically < 1% of the file, that's probably cheap enough that it won't 
matter too much. You could also probably set the upper limit on the volume 
size, because past a certain point the file changes will be limited by 
disk speeds, so from there on a bigger file doesn't imply more log space 
is required.

>> A (shadow volume) log should, ideally, also keep
>> additional sanity check information such as file
>> metadata (timestamps, size) for cross-check of
>> whether something went weird and the file was
>> changed underneath GlusterFS, and if it has, flush
>> out the log and force a full resync on the file.
>
> Hmm, this seems like an additional layer that might be
> nice (and perhaps an XML log would be appropriate
> here), but I would put it an separate inline
> translator so that it is not required.  The nice part
> is that if the protocol is extended to handle the
> journal layer, adding another separate layer like this
> would probably be easy!

For the sake of an extra few bytes in the log entry (8 byte time stamp + 8 
byte file size), I think it is probably worthwhile having it for 
crosscheck.

> Thanks again for your patience, I know it's not easy
> listening to back seat designers :)

I second that apology. :-)

Gordan