Allow Inconsistent Read Operations

[tbg]

For performance reasons, clients may want to carry out read-only operations that aren't concerned with consistency at all. Instead, they may simply want the data present at the replica connected to handed back to them, without going through consensus. Variations on this are possible, but the most straightforward route seems to be:

• adding optional stale semantics to RequestHeader via a boolean stale flag.
• returning an error when the stale flag is found on requests that alter state or within a Transaction.
• treating read-only requests tagged as stale specially: They do not alter the read timestamp cache, and they go straight to a mode of MVCC read that doesn't care about intents or uncertainty restarts but simply reads the first honest value it finds (either starting from the newest value, or from the given read timestamp). This involves carefully going through all the read operations individually, seeing where they converge and sorting out potential oddities. Possible we'll want to pass the 'stale' flag down to the MVCC operations where I think we're only using Scan and Get.
• making sure that we don't auto-wrap stale reads in a transaction internally when they span across ranges (see TxnCoordSender#sendOne).

We will also want to

• use this new type of read for meta[1,2] addressing records to use those stale lookups as suggested by @spencerkimball in Multiraft: leader leases #230 and
• check if there are other internal reads where we may want to use this.

[spencerkimball]

Instead of "stale", how about an end value called "readConsistency"? Calling it stale sort of implies it'll be stale when often it won't. Further there are at least three values, though we'll probably only support two for time being. There are:

CONSISTENT
INCONSISTENT

(For now)

We may want to add

CONSENSUS

In order to insist that reads go through consensus instead of relying on leases.

[tbg]

Makes sense except that I don't know why you, as a client, would want to explicitly force something through consensus. Even if that were faster in some tricky cases, you would have to be so involved in the internals to know when to use it. What's your reason for wanting to expose it?

[spencerkimball]

Like I said, I don't think it's high priority. Except various do implementations have added it as a failsafe when testing against jepsen and the like. You could easily slow down a failed leader's clock if you were being adversarial and convince it it still had a leader lease when in fact it didn't. Forcing through consensus is the gold standard.

[tbg]

You're right, it should be in there.