[Gluster-users] reading from local replica?

[Jeff Darcy]

> Am I misunderstanding cluster.read-subvolume/cluster.read-subvolume-index?
> 
> I have two regions, "A" and "B" with servers "a" and "b" in,
> respectfully, each region.  I have clients in both regions. Intra-region
> communication is fast, but the pipe between the regions is terrible.
> I'd like to minimize inter-region communication to as close to glusterfs
> write operations only and have reads go to the server in the region the
> client is running in.
> 
> I have created a replica volume as:
> gluster volume create gv0 replica 2 a:/data/brick1/gv0
> b:/data/brick1/gv0 force
> 
> As a baseline, if I use scp to copy from the brick directly, I get --
> for a 100M file -- times of about 6s if the client scps from the server
> in the same region and anywhere from 3 to 5 minutes if I the client scps
> the server in the other region.
> 
> I was under the impression (from something I read but can't now find)
> that glusterfs automatically picks the fastest replica, but that has not
> been my experience; glusterfs seems to generally prefer the server in
> the other region over the "local" one, with times usually in excess of 4
> minutes.

The choice of which replica to read from has become rather complicated
over time.  The first parameter that matters is cluster.read-hash-mode,
which selects between dynamic and (two forms of) static selection.  For
the default mode, we try to spread the read load across replicas based
on both the file's ID and the client's.  For read-hash-mode=0 *only*,
we do this.

 * If "choose-local" is set (as it is by default) and there's a local
   replica, use that.

 * Otherwise, select a replica based on fastest *initial* response.

Note that these are both a bit prone to hot spots, which is why this
method is not the default.  Also, re-evaluating response times is as
likely to lead to "mobile hotspot" behavior as anything else -
clients keep following each other around to previously idle but now
overloaded replicas, moving the congestion around but never resolving
it.  Thus, we only tend to re-evaluate in response to brick up/down
events.  Probably some room for improvement here.

That brings us to read-subvolume and read-subvolume-index.  The
difference between them is that read-subvolume takes a translator
*name* (which you'd have to get from the volfile) and only applies
to one replica set within a volume.  It's really only useful for
testing and debugging.  By contrast, read-subvolume-index applies
to all replica sets in a volume and doesn't require any knowledge
of translator names.  Either one is used *before* read-hash-mode;
if it's set, and if the corresponding replica is up, it will be
chosen.

Yes, it's a bit of a mess.  However, as you've clearly guessed,
this is a pretty critical decision so it's nice to have many
different ways to control it.

> I've also tried having clients mount the volume using the "xlator"
> options cluster.read-subvolume and cluster.read-subvolume-index, but
> neither seem to have any impact.  Here are sample mount commands to show
> what I'm attempting:
> 
> mount -t glusterfs -o xlator-option=cluster.read-subvolume=gv0-client-<0
> or 1> a:/gv0 /mnt/glusterfs
> mount -t glusterfs -o xlator-option=cluster.read-subvolume-index=<0 or
> 1> a:/gv0 /mnt/glusterfs

I would guess that the translator options are somehow not being passed
all the way through to the translator that actually makes the decision.
If it is being passed, it definitely should "force the decision" as
described above.  There might be a bug here, or perhaps I'm just
misunderstanding code I haven't read in a while.

Also, please not that synchronous replication (AFR) isn't really
intended or expected to work over long distances.  Anything over 5ms
RTT is risky territory; that's why we have separate geo-replication.