client: retryable Txn errors should require acknowledgment from clients

[nvanbenschoten]

See #22337 (comment).

During retryable error handling, a client.Txn will reset its state to prepare for a retry internally before returning the error that triggered the retry. When doing this, the transaction will either increment its epoch or replace its ID. Either way, the transaction throws out all work it has previously performed*.
After this point, the new incarnation of the transaction can be used just like it is a new transaction.

This allows for abuses of db.Txn and txn.Exec, where a retry loop does not restart on a retryable error. For instance, in the following example, if the second txn.Put fails with a retryable error, the transaction will finish without either of the Put operations' results being committed.

err := db.Txn(func (ctx context.Context, txn *client.Txn) error {
  if err := txn.Put(ctx, "key", "val"); err != nil {
    return err
  }
  // Don't really care if this succeeds.
  _ := txn.Put(...)
  return nil
})

In order to avoid this type of misuse, we should require some kind of buy-in from users of client.Txn before a retried transaction can be used again. This could be as simple as requiring PrepareForRetry to be called before the next operation. This is already called in the db.Txn retry loop when it sees retryable Txn errors, so it should be invisible in almost all cases where client.Txn is being used correctly.

This will be addressed partially by spencerkimball@9decb16, which forces all txn aborted errors to propagate up to the db.Exec retry loop.

[*] this is not completely true, as a transaction with an incremented epoch will not need to write a new txn record and will still keep track of previous intents, but that is all transparent to users of client.Txn.

[tbg]

@andreimatei @nvanbenschoten is this still 2.1 material?

[nvanbenschoten]

This is a part of @andreimatei's upcoming refactoring of the client.Txn api, so I'll defer to him on that.