rangefeed: add non-fatal assertion for intent commit resolved ts

[erikgrinaker]

This patch adds a non-fatal assertion for intent commit operations that are emitted below the resolved timestamp. A known bug will do just this, so the assertion is kept non-fatal to avoid disruption, but it will confirm whether a cluster is hitting the bug.

The log messages are of the form:

E240123 10:49:00.052256 57 kv/kvserver/rangefeed/resolved_timestamp.go:162  [n1,s1,r7/1:‹/Table/{3-4}›,rangefeed] 1  resolved timestamp 0.000000024,0 equal to or above timestamp of operation txn_id:a55c0992-581b-4cb2-97bf-fd9e22f7a657 key:"foo" timestamp:<wall_time:45 > value:"bar"

Touches #104309.
Touches #117612.
Epic: none
Release note: None

[cockroach-teamcity]

This change is 

[nvanbenschoten]

Reviewed 3 of 3 files at r1, all commit messages.
Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @erikgrinaker)

---

pkg/kv/kvserver/rangefeed/resolved_timestamp.go line 161 at r1 (raw file):

		// A known bug will trip this assertion, so don't fatal to avoid disruption.
		// See: https://github.com/cockroachdb/cockroach/issues/104309
		if t.Timestamp.LessEq(rts.resolvedTS) {

Should we expand assertOpAboveRTS to take a context and a fatal bool flag, then have it call either log.Fatalf or log.Errorf accordingly, then just call that here?

[erikgrinaker]

Reviewable status: complete! 0 of 0 LGTMs obtained (waiting on @nvanbenschoten)

---

pkg/kv/kvserver/rangefeed/resolved_timestamp.go line 161 at r1 (raw file):

Previously, nvanbenschoten (Nathan VanBenschoten) wrote…

Should we expand assertOpAboveRTS to take a context and a fatal bool flag, then have it call either log.Fatalf or log.Errorf accordingly, then just call that here?

If we were going to actually merge this, then yes. Since this is a one-off custom build, I'll just keep it as is.

Will address this over on #117612.