changefeed stucks after injecting TiKV failure

[fubinzh]

What did you do?

1. create redo log changefeed

cdc  cli  changefeed  create "--server=127.0.0.1:8301"  "--sink-uri=mysql://root:@downstream.cdc-testbed-tps-1892534-1-706:3306"  "--changefeed-id=redo-enable-cdc-all-node-restart-sync"  "--config=/tmp/changefeed.toml"

$ cat /tmp/changefeed.toml 
[consistent]
level = "eventual"
nstorage = "s3://tmp/test-infra-redolog/redo-enable-cdc-all-node-restart-sync9ac9a259-7f35-4e58-926d-0d6d216693a9?access-key=minioadmin&secret-access-key=minioadmin&endpoint=http://minio-peer:9000&force-path-style=true
max-log-size = 64

2. run workload
3. Injact failure for all TiKV

[2023/08/16 19:45:41.280 +00:00] [INFO] [step.go:44] ["kvFailure: duration=1m0s,  allTiKV=true, failNode=0"]
[2023/08/16 19:45:41.280 +00:00] [INFO] [tikv_chaos.go:44] ["Inject TiKV failure for all nodes"]
[2023/08/16 19:46:41.385 +00:00] [INFO] [chaos.go:93] ["run chaos finished"]

What did you expect to see?

Changefeed should not stuck after TiKV failure restored

What did you see instead?

redo log changefeed stucks

"[meta_manager.go:334] [\"Redo meta has not changed for a long time, owner may be stuck\"] [namespace=default] [changefeed=redo-enable-cdc-all-node-restart-sync] [lastFlushTime=9m21.99436182s] [meta=\"{\\\"CheckpointTs\\\":443604045771833536,\\\"ResolvedTs\\\":443604045771833536}\"]"

Versions of the cluster

CDC version:

"[version.go:47] [\"Welcome to Change Data Capture (CDC)\"] [release-version=v7.4.0-alpha] [git-hash=dcfcb43a99bacd7639156421993a3a95284c5f45] [git-branch=heads/refs/tags/v7.4.0-alpha] [utc-build-time=\"2023-08-16 11:36:28\"] [go-version=\"go version go1.21.0 linux/amd64\"] [failpoint-build=false]"

[nongfushanquan]

/assign @CharlesCheung96

[hicqu]

It can be reproduced in my local environment, even if without redo-log enabled.
It disappears after I revert #9519 , so I guess it's introduced by #9519 .

[fubinzh]

This issue also seen when running mysql sink changefeed and injecting tikv failures. (case: tikv_unavailable_sync)

[fubinzh]

/found automation
/severity critical

[fubinzh]

This issue also seen in case of tikv network partition or cdc scale scenarios.