Etcd not respond to progress notification on newly created watch RPC until there is a event in a watch

[serathius]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via [email protected].
• This is not a support request or question, support requests or questions should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

When trying make K8s depend on progress notification I found an broken edge case when working on kubernetes/kubernetes#123513

What I found is that #15237 introduced a bug with deferredProgress logic.

If a progress request is send when watch is not synced (like when it's just newly created watch), etcd mark progress notification as deferred, and stop responding to progress notification request.

Progress notification can be later restored but only if there is an event to be sent. So if there is no event observed in the watch, etcd will never respond to progress notification requests.

What did you expect to happen?

Don't expect event to arrive, send the progress notification when watchers become synced.

How can we reproduce it (as minimally and precisely as possible)?

Take PR.

Run make test WHAT=./vendor/k8s.io/apiserver/pkg/storage/... GOFLAGS="-v" KUBE_TEST_ARGS='--count 10 --run TestListWithListFromCache --failfast'

Anything else we need to know?

No response

Etcd version (please run commands below)

Reproduced on v3.5.10 embedded server

Etcd configuration (command line flags or environment variables)

paste your configuration here

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
# paste output here

$ etcdctl --endpoints=<member list> endpoint status -w table
# paste output here

Relevant log output

No response

[serathius]

Note, I don't expect this to be a super impactful thing, however definitely a blocker for Kubernetes to adopt and depend on manually requested progress notifications.

[ahrtr]

What I found is that #15237 introduced a bug with deferredProgress logic.

If a progress request is send when watch is not synced (like when it's just newly created watch), etcd mark progress notification as deferred, and stop responding to progress notification request.

Did not dig into the source code yet. But based on your input, It should be expected behaviour. I do not see any issue here. The progress notification events shouldn't be sent out until all events up to a revision have been already delivered. It's exactly what the Bookmarkable means. Please let me know if I misunderstood your point.

Take PR.

Run make test WHAT=./vendor/k8s.io/apiserver/pkg/storage/... GOFLAGS="-v" KUBE_TEST_ARGS='--count 10 --run TestListWithListFromCache --failfast'

Please do not expect every etcd contributor is familiar with K8s test case or has time to dig into K8s test case. Please create an dedicated etcd test case to clearly clarify the issue if you still think it's an etcd issue.

[ahrtr]

Can we close this ticket?

[serathius]

No, this is a big issue that blocks usage of manual progress notification in K8s.

Will work on this when I fix kubernetes/kubernetes#123532

[ahrtr]

No, this is a big issue that blocks usage of manual progress notification in K8s.

What issue? Let me know if you have comment on #17507 (comment)

[serathius]

Etcd not respond to progress notification on newly created watch RPC until there is a event in a watch

[ahrtr]

I already read your description, as I mentioned in #17507 (comment), it's expected behaviour.

I do not see any issue here. The progress notification events shouldn't be sent out until all events up to a revision have been already delivered. It's exactly what the Bookmarkable means. Please let me know if I misunderstood your point.