dm-worker hang when etcd watch ch closed

[Ehco1996]

What did you do?

one dm-worker node in weak network environment

so thread of WatchSourceBound exit by

tiflow/dm/pkg/ha/bound.go

Line 273 in 1c1015b

return

How did I determine that exited on this line？ in logs we can see

[2022/01/03 15:49:16.622 +08:00] [INFO] [server.go:581] ["receive source bound"] [bound="{\"source\":\"\",\"worker\":\"dm-xxxx\"}"] ["is deleted"=true]

do some unboud source operateions

[2022/01/03 16:22:30.662 +08:00] [INFO] [server.go:602] ["handleSourceBound will quit now"]

because there is no error log like observeSourceBound is failed and will quit now, se we can make sure handleSourceBound exit becasue sourceBoundCh or sourceBoundErrCh is closed

but the thread of keepalive( keepalive and watchsourceBound may use different grpc stream ) is still working...

[2022/01/03 16:22:31.663 +08:00] [INFO] [join.go:93] ["start to keepalive with master"]
[2022/01/03 16:24:49.471 +08:00] [INFO] [keepalive.go:139] ["keep alive channel is closed"]
[2022/01/03 16:24:51.517 +08:00] [WARN] [keepalive.go:122] ["fail to revoke lease"] []
[2022/01/03 16:24:51.517 +08:00] [WARN] [join.go:105] ["keepalive with master goroutine paused"] []
[2022/01/03 16:24:51.517 +08:00] [WARN] [server.go:558] ["worker has not been started, no need to stop"] [source=]
[2022/01/03 16:24:52.517 +08:00] [INFO] [join.go:93] ["start to keepalive with master"]

as a result, dm-master think this worker is alive ,but actually the worker is hang forever

how to reproduce this?

see #4553

What did you expect to see?

dm-worker offline when observeSourceBound quit

What did you see instead?

so the response of dmctl query-status is:

❯ dmctl query-status
{
    "result": true,
    "msg": "",
    "sources": [
        {
            "result": false,
            "msg": "[code=40070:class=dm-worker:scope=internal:level=high], Message: no mysql source is being handled in the worker",
            "sourceStatus": {
                "source": "mysql-replica-01",
                "worker": "worker1",
                "result": null,
                "relayStatus": null
            },
            "subTaskStatus": [
            ]
        }
    ]
}

Versions of the cluster

master

current status of DM cluster (execute query-status <task-name> in dmctl)

(paste current status of DM cluster here)

[Ehco1996]

@XuJianxu i am not sure about this bug's level could you please add a lable for me?

[Ehco1996]

close for no updated now, see more here