storage: introduce IndeterminateCommitError and RecoverTxnRequest

This PR polishes off the recovery mechanism from #35165 and gets it into
a reviewable state. The mechanism comes in three main parts. The first is
the addition of a new STAGING transaction status. This status is not
reachable yet through standard KV requests, but it triggers parallel commit
recovery and is injected in tests.

The next part is related to `PushTxn`, which is used both in concurrency control
and garbage collection. Pushing a STAGING transaction acts as expected if the
push fails. However, if the push succeeds, it is not allowed to directly modify
the STAGING transaction record. Instead, it now returns a new
`IndeterminateCommitError`. This error is captured by a store and triggers a new
transaction recovery mechanism, which is the third part of this change.

The transaction recovery mechanism queries all of the STAGING transaction's
in-flight writes using QueryIntent(IfMissing=PREVENT) requests. The process then
passes whether or not it found all declared writes to a new RecoverTxnRequest
that takes care of recovering the transaction record from the STAGING state to
either a COMMITTED or ABORTED state.

Take a look at `txn_recovery_integration_test.go` for a end-to-end integration
test of this recovery mechanism.

Release note: None

docs/RFCS/20180324_parallel_commit.md

@@ -1,5 +1,5 @@
 - Feature Name: parallel commit
-- Status: draft
+- Status: in-progress
 - Start Date: 2018-03-24
 - Authors: Tobias Schottdorf, Nathan VanBenschoten
 - RFC PR: #24194

pkg/roachpb/api.go

@@ -507,6 +507,9 @@ func (*GCRequest) Method() Method { return GC }
 // Method implements the Request interface.
 func (*PushTxnRequest) Method() Method { return PushTxn }
 
+// Method implements the Request interface.
+func (*RecoverTxnRequest) Method() Method { return RecoverTxn }
+
 // Method implements the Request interface.
 func (*QueryTxnRequest) Method() Method { return QueryTxn }
 
@@ -693,6 +696,12 @@ func (ptr *PushTxnRequest) ShallowCopy() Request {
 	return &shallowCopy
 }
 
+// ShallowCopy implements the Request interface.
+func (rtr *RecoverTxnRequest) ShallowCopy() Request {
+	shallowCopy := *rtr
+	return &shallowCopy
+}
+
 // ShallowCopy implements the Request interface.
 func (qtr *QueryTxnRequest) ShallowCopy() Request {
 	shallowCopy := *qtr
@@ -1021,7 +1030,8 @@ func (*GCRequest) flags() int                  { return isWrite | isRange }
 func (*PushTxnRequest) flags() int {
 	return isWrite | isAlone | updatesReadTSCache | updatesWriteTSCache
 }
-func (*QueryTxnRequest) flags() int { return isRead | isAlone }
+func (*RecoverTxnRequest) flags() int { return isWrite | isAlone | updatesWriteTSCache }
+func (*QueryTxnRequest) flags() int   { return isRead | isAlone }
 
 // QueryIntent only updates the read timestamp cache when attempting
 // to prevent an intent that is found missing from ever being written