Log errors when polling for the ES version

[afharo]

In pollEsNodesVersion, we are requesting the version every healthCheckInterval (defaults to 2.5s).

I think we should log when ES returns an error in this piece of logic. This validation can halt Kibana when an error connecting to ES occurs (both, during startup it holds SavedObjects migrations from happening, holding the rest of Kibana startup process, and during normal operations because the status will go RED and many services depending on it will fail). I'd say it'll be helpful if we log.error these to help with the troubleshooting.

What do you think?

Scope:

• Add logging to ES errors before the first successful connection in the ES status check
• Logging should be throttled (every 5 minutes) as to not cause very noisy logs

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[pgayvallet]

This means that, if the responses from ES take longer than that, we'll pile up requests, potentially killing Kibana due to an OOM.

I'm not sure if that's true, as we're using an exhaustMap operator which is supposed to ignore further projections until the current projection is completed (but I'm no rxjs expert)

exhaustMap
Returns an Observable that emits items based on applying a function that you supply to each item emitted by the source Observable, where that function returns an (so-called “inner”) Observable. When it projects a source value to an Observable, the output Observable begins emitting the items emitted by that projected Observable. However, exhaustMap ignores every new projected Observable if the previous projected Observable has not yet completed. Once that one completes, it will accept and flatten the next projected Observable and repeat this process.

kibana/src/core/server/elasticsearch/version_check/ensure_es_version.ts

Lines 136 to 153 in 4584a8b

	return timer(0, healthCheckInterval).pipe(
	exhaustMap(() => {
	return from(
	internalClient.nodes.info<NodesInfo>({
	filter_path: ['nodes.*.version', 'nodes.*.http.publish_address', 'nodes.*.ip'],
	})
	).pipe(
	map(({ body }) => body),
	catchError((_err) => {
	return of({ nodes: {} });
	})
	);
	}),
	map((nodesInfo: NodesInfo) =>
	mapNodesVersionCompatibility(nodesInfo, kibanaVersion, ignoreVersionMismatch)
	),
	distinctUntilChanged(compareNodes) // Only emit if there are new nodes or versions
	);

[afharo]

You're right! The exhaustMap holds any other emissions, so we're not accumulating requests.

However, I ran a local test with the healthCheckInterval set to 100ms and mocking ES to hold the requests for 10x that, and while I don't see new requests coming in, the memory grows (and releases after timeouts). Probably the timer's emitted values are enqueued? 🤷

In any case, I think the log entry would be much welcomed. I'll update the description. Thanks for pointing that out @pgayvallet