redo log writer runGC failed because of can't read log file directory

[amyangfei]

What did you do?

1. Create a tidb cluster, start up a TiCDC
2. create a changefeed by cdc cli changefeed create --pd=http://127.0.0.1:2379 -c bypass-cf-5 --sink-uri="blackhole://" --config changefeed5_nfs.toml
3. changefeed5_nfs.toml

[consistent]
level = "eventual"
storage = "nfs:///home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5"

Path /home/apple/work/code/pingcap/ticdc/bin/redo/ exists, but /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5 doesn't.

What did you expect to see?

cdc runs normally

What did you see instead?

cdc keeps logging

[2021/11/18 17:17:52.891 +08:00] [ERROR] [writer.go:237] ["redo log GC fail"] [changefeedID=bypass-cf-5] [error="[CDC:ErrRedoFileOp]can't read log file directory: /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: open /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: no such file or directory: can't read log file directory: /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: open /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: no such file or directory; [CDC:ErrRedoFileOp]can't read log file directory: /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: open /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: no such file or directory: can't read log file directory: /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: open /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: no such file or directory"] [errorVerbose="the following errors occurred:\n -  [CDC:ErrRedoFileOp]can't read log file directory: /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: open /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: no such file or directory: can't read log file directory: /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: open /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: no such file or directory\n    github.com/pingcap/errors.AddStack\n    \tgithub.com/pingcap/[email protected]/errors.go:174\n    github.com/pingcap/errors.(*Error).GenWithStackByCause\n    \tgithub.com/pingcap/[email protected]/normalize.go:302\n    github.com/pingcap/ticdc/pkg/errors.WrapError\n    \tgithub.com/pingcap/ticdc/pkg/errors/helper.go:30\n    github.com/pingcap/ticdc/cdc/redo/writer.(*Writer).getShouldRemovedFiles\n    \tgithub.com/pingcap/ticdc/cdc/redo/writer/file.go:462\n    github.com/pingcap/ticdc/cdc/redo/writer.(*Writer).GC\n    \tgithub.com/pingcap/ticdc/cdc/redo/writer/file.go:411\n    github.com/pingcap/ticdc/cdc/redo/writer.(*LogWriter).gc\n    \tgithub.com/pingcap/ticdc/cdc/redo/writer/writer.go:249\n    github.com/pingcap/ticdc/cdc/redo/writer.(*LogWriter).runGC\n    \tgithub.com/pingcap/ticdc/cdc/redo/writer/writer.go:235\n    runtime.goexit\n    \truntime/asm_amd64.s:1371\n -  [CDC:ErrRedoFileOp]can't read log file directory: /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: open /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: no such file or directory: can't read log file directory: /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: open /home/apple/work/code/pingcap/ticdc/bin/redo/bypass-cf-5: no such file or directory\n    github.com/pingcap/errors.AddStack\n    \tgithub.com/pingcap/[email protected]/errors.go:174\n    github.com/pingcap/errors.(*Error).GenWithStackByCause\n    \tgithub.com/pingcap/[email protected]/normalize.go:302\n    github.com/pingcap/ticdc/pkg/errors.WrapError\n    \tgithub.com/pingcap/ticdc/pkg/errors/helper.go:30\n    github.com/pingcap/ticdc/cdc/redo/writer.(*Writer).getShouldRemovedFiles\n    \tgithub.com/pingcap/ticdc/cdc/redo/writer/file.go:462\n    github.com/pingcap/ticdc/cdc/redo/writer.(*Writer).GC\n    \tgithub.com/pingcap/ticdc/cdc/redo/writer/file.go:411\n    github.com/pingcap/ticdc/cdc/redo/writer.(*LogWriter).gc\n    \tgithub.com/pingcap/ticdc/cdc/redo/writer/writer.go:250\n    github.com/pingcap/ticdc/cdc/redo/writer.(*LogWriter).runGC\n    \tgithub.com/pingcap/ticdc/cdc/redo/writer/writer.go:235\n    runtime.goexit\n   
 \truntime/asm_amd64.s:1371"]

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

v5.3.0

TiCDC version (execute cdc version):

master@pingcap/ticdc@fe92b89