kvserver: properly handle DeleteRange in batches that hit WriteTooOld errors

[AlexTalks]

Currently, batches that have DeleteRange may read a different set of keys to delete if they have subsequent blind writes that hit a WriteTooOld error (and leave intents behind). If this happens and one of the keys that need to be deleted at the refreshed timestamp already has an intent on it, we won't be able to replay the batch as an intent that should be generated by the DeleteRange won't be present in the intent history.

This manifests in errors such as:

transaction b81df504-9bf5-4e2b-9a94-f9b19cee01ab with sequence 2 missing an intent with lower sequence 1

An example batch that could hit this error is below, assuming the Put operation hits a WriteTooOld error:

b := &Batch{}
b.DelRange(ctx, /Table/50/"006f9bc1", /Table/50/"32de5dba", true)
b.Put(ctx, /Table/50/"130f043b", v-1)
txn.Run(ctx, b)

This was discovered upon adding DeleteRange to kvnemesis in #68003

Jira issue: CRDB-10452

[nvanbenschoten]

@AlexTalks and I discussed what to do about this. KV batches are never used in a way that hits this currently, and the error already means that this won't introduce silent corruption, so there's not a lot of urgency to support this. Instead, we decided to update the error message to point at this issue as a possible suspect and then place the resolution of this on hold.

We also decided that when updating the error message, we'll update the errors in replayTransactionalWrite from errors.Errorf to errors.AssertionFailedf.

[mwang1026]

If the plan is to update an error message, can we pick that up quickly and close this out?

[AlexTalks]

If the plan is to update an error message, can we pick that up quickly and close this out?

The update to the error message is in #73496, however I believe that @nvanbenschoten and I discussed leaving this issue open as a reference to the actual issue here (which is that DelRange operations in co-mingled batches are not supported), and linking here in the error message. If it's okay to close this out, I'll remove the issue link and we can do that however.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!