Informer frequent desyncs on slow-moving resources in <= 1.15

[clux]

Need to revisit the choice made in #134. In particular the choice discussed here:

There are two different options when desyncs happen:

1. Do a LIST to get the latest resourceVersion, losing all events between the desync and the LIST
2. Always set resourceVersion to "0", generating WatchEvent::Added events for all resources that already exist

We went with option 2, back then and it generally works fine... but..

...if we have an informer on a namespace with particularly slow-moving resources, then choice 2 can cause us to reset the resourceVersion to 0 every poll after the first one (because the last returned resourceVersion is still too old for a new watch call). (example will link to this in a few).

Ran into this in #218 when trying to see if we could just use the Informer's logic to drive a Reflector to simplify some code. The list call + watch from that is what saves us from ever encountering this with the current Reflector. Unfortunately, Informer as it stands is actually unusable on certain pathological cases.

Maybe, we should consider having Informer do a list call after all. It's possible to bunch-up the data returned as WatchEvents to emulate the current "start from zero" behaviour..

[clux]

Related issue: kubernetes-client/python#819

[nightkr]

Or, we could implement support for watch bookmarks (https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/20190206-watch-bookmark.md), which makes it the apiserver's responsibility to send the client the new resourceVersion before the old one is invalidated.

[clux]

Have implemented this now in the same branch, but it's not much use until everyone's got at least kubernetes 1.16 (where it first became beta)

[nightkr]

I don't think that's unreasonable, given that 1.15 is EOL now. And if it's a major problem for them then they can turn on the WatchBookmark feature gate.

[clux]

What's EOL on their side isn't what's readily available though:

• EKS range: 1.12, 1.13, 1.14, 1.15
• GCP 1.14 stable -> 1.15 regular -> 1.16 rapid
• AKS.. harder to tell, but looks like they're just starting to get 1.16

We are pretty close, but still some time to go. I still rely on EKS and would like Reflectors working. Though perhaps a few extra watch calls every 300s isn't such a bad hit to help incentivise people to upgrade 🤔

[nightkr]

Oh, ouch. But yeah, it's not like anything is /broken/ on 1.15. Sadly it doesn't look like EKS allows you to customize the feature gates either.. aws/containers-roadmap#487 aws/containers-roadmap#512

[clux]

Yeah, I guess it's not broken-broken. It's a network inefficiency (which will probably last for months for consumers).

Even if we don't do the part of #218 that makes Reflector's use Informers internally yet (which is such a simplification), there's still the issue of informers themselves having the same issue unless we start porting the reflector list-then-watch functionality into the informer.

Still with EKS probably getting it soon™, maybe we just mark the next kube version's runtime module as a "works best with 1.16 and above". It's not like it's been particularly stable in the past two months anyway.

[clux]

Fixed it in Reflectors by reverting the change for now. Probably wontfix Informer. Put a recommended with kubernetes >= 1.16 on it.

[nightkr]

FWIW I'm not sure I understand how Reflector could do a better job on its own than Informer. The three possible reactions would be:

1. Clear the cache, set resourceVersion=0, restart watch
   • Informer's approach
2. Reset cache and resourceVersion from a LIST call, restart watch
3. Do a LIST with an empty filter go get a recent resourceVersion, restart watch

1 and 2 should be pretty much equivalent, since you still need to download all of the objects. LIST might have slightly less overhead per item, while rewatching from 0 avoids an API call.

3 is broken-broken, since it will lose writes and get stuck with stale objects.

[clux]

I think approach 2 used by Reflector is better than 1 because the resourceVersion returned by the LIST call is a dynamic value, whereas the last watchevent's resourceVersion is the resourceVersion of the last object when it last updated.

[nightkr]

True, that's a good point.

[clux]

Unmarking as a bug. It's kind of documented behaviour of the pre-bookmark world.

People are reporting it's not a thing with bookmarks, preliminarily, but haven't been able to fully test it out myself yet.