schedule reconnect in case of HTTP_GONE

[honnix]

Reset resource version to null and schedule reconnect in case of HTTP_GHONE

This tries to address #1529

[centos-ci]

Can one of the admins verify this patch?

[honnix]

Couldn't figure out how this could fail PATCH request. 🤔

[rohanKanojia]

ok to test

[rohanKanojia]

@honnix : Not your fault. Our tests are flaky.

[honnix]

I seems I should fix these test cases as they look very relevant.

[rohanKanojia]

I meant integration tests, not unit tests

[honnix]

@rohanKanojia Yeah I'm working on it. Not very familiar with the code base, so I didn't find the test case.

[honnix]

Now this makes me wondering whether watching from the beginning of history is a bad idea. Can someone point me how resourceVersion is set initially before creating a WatchConnectionManager?

[honnix]

This file is not named properly and doesn't seem to be picked up by junit.

[honnix]

Maybe it is ok because when this happens, old resource versions got wiped out from history.

[honnix]

@rohanKanojia PTAL again. Thanks.

[rohanKanojia]

@honnix : Could you please add a line to CHANGELOG regarding this change?

[honnix]

@rohanKanojia Sure. I saw that label but didn't realise I should update it. Will do.

[rohanKanojia]

[merge]

[wiggzz]

Thanks - appreciate the quick turn around.

[honnix]

Likewise, thanks for pointing out the defect.

[rohanKanojia]

@honnix @wiggzz : Thanks guys, I will cut a release soon this week and let your guys know

[xiateapu]

I'm a novice, so.... could you please tell me when or why kubernetes api returns HTTP_GONE?

[wiggzz]

I'm a novice, so.... could you please tell me when or why kubernetes api returns HTTP_GONE?

In the context of a watch, it will return HTTP_GONE when you ask to see changes for a resourceVersion that is too old - i.e. when it can no longer tell you what has changed since that version, since too many things have changed. In that case, you'll need to start again, by not specifying a resourceVersion in which case the watch will send you the current state of the thing you are watching and then send updates from that point.

[rohanKanojia]

@xiateapu : ^^

[xiateapu]

Actually i had this problem as shown below:

It seems that resourceVersion has a big gap between last watched and current ?
So i'm very confused that why dose the too many things have changed happened？Can‘t kubernetes tell me these change in time?

[xiateapu]

I'm a novice, so.... could you please tell me when or why kubernetes api returns HTTP_GONE?

In the context of a watch, it will return HTTP_GONE when you ask to see changes for a resourceVersion that is too old - i.e. when it can no longer tell you what has changed since that version, since too many things have changed. In that case, you'll need to start again, by not specifying a resourceVersion in which case the watch will send you the current state of the thing you are watching and then send updates from that point.

Thanks for your answer ^ ^

[xiateapu]

@xiateapu : ^^

^ ^

[wind57]

I'm a novice, so.... could you please tell me when or why kubernetes api returns HTTP_GONE?

In the context of a watch, it will return HTTP_GONE when you ask to see changes for a resourceVersion that is too old - i.e. when it can no longer tell you what has changed since that version, since too many things have changed. In that case, you'll need to start again, by not specifying a resourceVersion in which case the watch will send you the current state of the thing you are watching and then send updates from that point.

what about an informer? ResourceEventHandler has a default method onNothing. Its documentation says the same thing :

Called after an empty list is retrieved on start or after an HTTP GONE when the Store is empty

I am asking this in the context of spring-cloud-kubernetes project. Previously, we had used Watches for detecting changes in config maps/secrets. I am trying to move to informers now. So our code is like this at the moment:

					@Override
					public void onClose(WatcherException exception) {
						log.warn("ConfigMaps watch closed", exception);
						Optional.ofNullable(exception).map(e -> {
							log.debug("Exception received during watch", e);
							return exception.asClientException();
						}).map(KubernetesClientException::getStatus).map(Status::getCode)
								.filter(c -> c.equals(HttpURLConnection.HTTP_GONE)).ifPresent(c -> watch());
					}

If I move this to informers, it seems that I should handle this in the same manner, something like:

				@Override
				public void onNothing() {
					boolean isStoreEmpty = informer.getStore().list().isEmpty();
					if(!isStoreEmpty) {
						// HTTP_GONE, thus re-inform
						inform();
					}
				}

I am a bit lost (due to my limited knowledge) between "this should be handled", or an informer (like a watcher) will know what to do on its own.

Can some of the pros comment on this please? thank you.

[manusa]

Hi @wind57
There have been a few issues on this repo about Spring Cloud Kubernetes and the HTTP Gone problem. With more recent versions of the client, this should be less of a problem since watchers are started using bookmarks (if the cluster supports it).

In any case, as suggested in some of the reported issues, Spring Cloud should migrate from using raw Watchers to Informers.

Informers are self-healing, so as a client consumer you should not worry about reconnecting, and so on. You can use the onNothing event to log or monitor these incidents, but you shouldn't do any error-recovery stuff.

For your use-case, you can consider Informers as improved Watchers which take care of all the stuff for you.

[wind57]

thank you @manusa. this is exactly what I am trying to do currently, move to informers, it's almost ready For the time being I will comment onNothing and see if we get any reports in the future.

As usual, much appreciate your answer.