TiFlash result is not correct in consistency test

[lilinghai]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

insert with currency 5 and diff result tikv with tiflash

[2024/03/11 12:08:54.808 +08:00] [INFO] [verify.go:163] ["verify successfully"] [tikv="[35771550]"] [tiflash="[35771550]"] [tag=]
[2024/03/11 12:08:54.854 +08:00] [ERROR] [consistency_test.go:666] ["check consistence error"] ["db name"=db12] [sql="select count(*) from other_handle7"] [realSql="select /*+ ignore_index(other_handle7,PRIMARY) */  count(*) from other_handle7"] [
"prev succ"=3143] [error="result inconsistent tso 448300353380155718, difference: row number: 1, row1: 35068980, row2: 35068960, tikv: [35068980], tiflash [35068960]"] [stack="github.com/pingcap/endless/testcase/tiflash/consistency_test.checkConsi
stency.func1\n\t/home/jenkins/agent/workspace/endless-master-build/testcase/tiflash/consistency/consistency_test.go:666\ngithub.com/pingcap/endless/pkg/util.WithGinkgoRecover.func1\n\t/go/pkg/mod/github.com/pingcap/[email protected]
047c8b63a1/pkg/util/fastfail.go:18\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiFlash version? (Required)

master

[CalvinNeo]

Maybe due to #3971 of reusing history read index results of the same region.
This is because if two txns are not visible to each other, the order of there commit record may not correspond to their start_ts. That means the following may happen, given start_ts1 < start_ts2 < commit_ts1 < commit_ts2

index 10: Put Write CF commit_ts2
index 11: Put Write CF commit_ts1

The TiFlash's SOTA strategy is when handled a read index of tso commit_ts2, the following read index of commit_ts1 will reuse the result of 10 rather than 11, which is wrong.
If so, it affects releases starting from 6.0.

[JaySon-Huang]

All versions before v8.1 are affected.

Abstracted reproduce steps:

• Thread A write some rows, naming batch_1. Then acquire a segment snapshot snap_a
• Thread C write some rows, naming batch_2, with exactly the same row_ids and version with the batch_1
• Thread B acquires a segment snapshot snap_b
• Thread B get executed before thread A and update the shared-delta-index of the segment
  • However, when thread B update the shared-delta-index, the rows in batch_1 get deleted
• Thread A continue the read and reuse the shared-delta-index, but it can not read the rows in batch_1 because the rows are deleted from shared-delta-index

This issue can automatically disappear in the following reads. Because following read will use a more advance delta-index with the delta covering the duplicated rows.