cdc panic: The commitTs of the emit row is less than the received row

[fubinzh]

What did you do?

1. Enable redo log when there is a cdc log of 2 hours.

What did you expect to see?

cdc should not panic

What did you see instead?

cdc panic

WARN] TiCDC server data-dir is not set. Please use `cdc server --data-dir` to start the cdc server if possible.
panic: The commitTs of the emit row is less than the received row

goroutine 1814 [running]:
[go.uber.org/zap/zapcore.CheckWriteAction.OnWrite](http://go.uber.org/zap/zapcore.CheckWriteAction.OnWrite)(0x2?, 0x2?, {0x0?, 0x0?, 0xc42a62cfc0?})
        [go.uber.org/[email protected]/zapcore/entry.go:198](http://go.uber.org/[email protected]/zapcore/entry.go:198) +0x65
[go.uber.org/zap/zapcore.(*CheckedEntry).Write](http://go.uber.org/zap/zapcore.(*CheckedEntry).Write)(0xc458746410, {0xc087113900, 0x2, 0x2})
        [go.uber.org/[email protected]/zapcore/entry.go:264](http://go.uber.org/[email protected]/zapcore/entry.go:264) +0x3ec
[go.uber.org/zap.(*Logger).Panic](http://go.uber.org/zap.(*Logger).Panic)(0xc0000b6a00?, {0x3adc597?, 0x414705?}, {0xc087113900, 0x2, 0x2})
        [go.uber.org/[email protected]/logger.go:251](http://go.uber.org/[email protected]/logger.go:251) +0x59
[github.com/pingcap/log.Panic](http://github.com/pingcap/log.Panic)({0x3adc597?, 0xc087113940?}, {0xc087113900?, 0xc370994780?, 0xc31cbe50b8?})
        [github.com/pingcap/[email protected]/global.go:54](http://github.com/pingcap/[email protected]/global.go:54) +0x8b
[github.com/pingcap/tiflow/cdc/sinkv2/eventsink.(*TxnEventAppender).Append](http://github.com/pingcap/tiflow/cdc/sinkv2/eventsink.(*TxnEventAppender).Append)(0xc220a13b90, {0xc39fc4b000?, 0xc0090e49c0?, 0x4?}, {0xc375
ccc000, 0x57, 0x46d86e?})
        [github.com/pingcap/tiflow/cdc/sinkv2/eventsink/event_appender.go:85](http://github.com/pingcap/tiflow/cdc/sinkv2/eventsink/event_appender.go:85) +0x589
[github.com/pingcap/tiflow/cdc/sinkv2/tablesink](http://github.com/pingcap/tiflow/cdc/sinkv2/tablesink).(*EventTableSink[...]).AppendRowChangedEvents(0xc1e57fa6e0?, {0xc375ccc000, 0x57?, 0x7
f789a73afa8})
        [github.com/pingcap/tiflow/cdc/sinkv2/tablesink/table_sink_impl.go:78](http://github.com/pingcap/tiflow/cdc/sinkv2/tablesink/table_sink_impl.go:78) +0x59
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*tableSinkWrapper).appendRowChangedEvents](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*tableSinkWrapper).appendRowChangedEvents)(0x9?, {0xc375ccc000?, 0x0?, 0xc2b19c67
e0?})
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_wrapper.go:177](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_wrapper.go:177) +0x177
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask.func5(0x0)](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask.func5(0x0))
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:228](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:228) +0xb6
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask.func6](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask.func6)(0x0, 0x1)
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:294](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:294) +0x44f
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTask)(0xc003cca770, {0x4313928, 0xc0090e0200}, 0xc36f8f38c0)
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:386](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:386) +0xee9
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTasks](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*sinkWorker).handleTasks)(0x0?, {0x4313928, 0xc0090e0200}, 0xc008a70b40)
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:75](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/table_sink_worker.go:75) +0xc5
[github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*SinkManager).startSinkWorkers.func1()](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager.(*SinkManager).startSinkWorkers.func1())
        [github.com/pingcap/tiflow/cdc/processor/sinkmanager/manager.go:409](http://github.com/pingcap/tiflow/cdc/processor/sinkmanager/manager.go:409) +0x30
[golang.org/x/sync/errgroup.(*Group).Go.func1()](http://golang.org/x/sync/errgroup.(*Group).Go.func1())
        [golang.org/x/[email protected]/errgroup/errgroup.go:75](http://golang.org/x/[email protected]/errgroup/errgroup.go:75) +0x64
created by [golang.org/x/sync/errgroup.(*Group).Go](http://golang.org/x/sync/errgroup.(*Group).Go)
        [golang.org/x/[email protected]/errgroup/errgroup.go:72](http://golang.org/x/[email protected]/errgroup/errgroup.go:72) +0xa5

Versions of the cluster

bash-5.1# /cdc version
Release Version: v6.5.5-20231109-c8eed98
Git Commit Hash: c8eed98
Git Branch: heads/refs/tags/v6.5.5-20231109-c8eed98
UTC Build Time: 2023-11-09 09:24:38
Go Version: go version go1.19.12 linux/amd64
Failpoint Build: false

[fubinzh]

/severity major