spring-cloud-kubernetes-fabric8-leader: 2.0.1 - Watcher stopped unexpectedly, will restart

[whgibbo]

Hi,
Using the spring-cloud-kubernetes-fabric8-leader: 2.0.1 to determine a leader for the services in my pod.

Noticed the following warnings appearing:

021-02-05 12:19:23,429 WARN  [] [OkHttp https://172.20.0.1/...] o.s.c.k.f.l.Fabric8PodReadinessWatcher.onClose 104: Watcher stopped unexpectedly, will restart
io.fabric8.kubernetes.client.KubernetesClientException: too old resource version: 58642569 (58743275)
        at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:257)
        at okhttp3.internal.ws.RealWebSocket.onReadMessage(RealWebSocket.kt:333)
        at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.kt:245)
        at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.kt:106)
        at okhttp3.internal.ws.RealWebSocket.loopReader(RealWebSocket.kt:293)
        at okhttp3.internal.ws.RealWebSocket$connect$1.onResponse(RealWebSocket.kt:195)
        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Any thoughts or solutions?

[rohanKanojia]

In past we had added reconnect on 410 but it was reverted, see #1800 (comment) . You should retry watch from client side.

[whgibbo]

@rohanKanojia If you see the leader example, it hides the watcher..

So any thoughts on how to achieve this ?

[stale]

This issue has been automatically marked as stale because it has not had any activity since 90 days. It will be closed if no further activity occurs within 7 days. Thank you for your contributions!

[echo-liang]

@rohanKanojia If you see the leader example, it hides the watcher..

So any thoughts on how to achieve this ?

Hi there, did you find any solution to that? I had similar issues, got those watcher warning showing up in the server log, wanted to avoid them.

[manusa]

So any thoughts on how to achieve this ?

The underlying implementation o Spring Cloud should be using a Shared Informer instead of a Watcher.

[steinwelberg]

See this StackOverflow post for a sample implementation: https://stackoverflow.com/questions/61409596/kubernetes-too-old-resource-version

[cstrombe]

Just chiming in also, am seeing this warning in logs continuously (about 3-4x per hour). Using spring-cloud-kubernetes-fabric8-leader version 2.1.0.

[manusa]

From the context I understand that this needs to be fixed downstream.

Is there an open issue there? Maybe we can help trying to port whatever the current implementation is to use a SharedInformer instead of a Watcher.

[shawkins]

Just chiming in also, am seeing this warning in logs continuously (about 3-4x per hour). Using spring-cloud-kubernetes-fabric8-leader version 2.1.0.

What version is that on? The latest versions of fabric8 also now support bookmarks, so you won't typically resource too old. Prior versions would routinely see resource too old after the watch connections timed out.

[manusa]

Just chiming in also, am seeing this warning in logs continuously (about 3-4x per hour). Using spring-cloud-kubernetes-fabric8-leader version 2.1.0.

What version is that on? The latest versions of fabric8 also now support bookmarks, so you won't typically resource too old. Prior versions would routinely see resource too old after the watch connections timed out.

5.9.0 https://github.com/spring-cloud/spring-cloud-kubernetes/blob/902789a6a76eb527d9a53559695c005e8aa390aa/spring-cloud-kubernetes-dependencies/pom.xml#L38

[shawkins]

That's before automatic bookmark support in 5.11 f45ed4f

But you could manually request bookmarks in 5.9 5b444d7

[cstrombe]

I added a dependency in my maven pom on kubernetes-client-bom 5.11.0 and have yet to see the warning in logs after deploying...

[shawkins]

Marking as resolved by bookmark support.