sql: inconsistent scan can return error cannot specify timestamp older than 5m0s for this operation if cluster is overloaded

[renatolabs]

The following error can observed in a cluster that is under a lot of load:

jobs/registry.go:1520 ⋮ [T1,n1] 761  AUTO CREATE STATS job 852625652397473796: stepping through state reverting with error: AS OF SYSTEM TIME: cannot specify timestamp older than 5m0s for this operation
jobs/registry.go:1520 ⋮ [T1,n1] 761 +(1) attached stack trace
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  -- stack trace:
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  | github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).StartInconsistentScan
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  |         github.com/cockroachdb/cockroach/pkg/sql/row/fetcher.go:580
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*tableReader).startScan
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  |         github.com/cockroachdb/cockroach/pkg/sql/rowexec/tablereader.go:219
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*tableReader).Next
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  |         github.com/cockroachdb/cockroach/pkg/sql/rowexec/tablereader.go:257
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*samplerProcessor).mainLoop
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  |         github.com/cockroachdb/cockroach/pkg/sql/rowexec/sampler.go:254
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*samplerProcessor).Run
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  |         github.com/cockroachdb/cockroach/pkg/sql/rowexec/sampler.go:228
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  | github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).StartInternal.func1
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  |         github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:510
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  | runtime.goexit
jobs/registry.go:1520 ⋮ [T1,n1] 761 +  |         GOROOT/src/runtime/asm_amd64.s:1594

This seems easy to reproduce by setting up a cluster and running tpcc with more warehouses than the cluster can comfortably handle. For reference, I saw a couple of occurrences of this error in a 4-node (n1-standard-4) cluster with 200 warehouses.

Likely not relevant: the error above happened while in a mixed-version state. I don't think this is related, but worth pointing out. It has also been seen in roachtests before (coincidentally or not, in another mixed version test): #93623 (comment)

Jira issue: CRDB-26361

[RaduBerinde]

The check in StartInconsistentScan that generates this error can just be removed I think. The txn will be recreated as needed by sendFn.

[mgartner]

@inata4 I see you added the support label. Do you remember what ticket this issue was related to?

[inata4]

I think we saw it in https://cockroachdb.zendesk.com/agent/tickets/21535