WIP [Discover] Migrate field existence to API request

[kertal]

Currently in Discover, we calculate the available fields by the data returned by Elasticsearch.

To show all fields users need to change the Hide empty fields filter

Here's where the field statistics are calculated:

kibana/src/plugins/discover/public/application/main/components/sidebar/discover_sidebar_responsive.tsx

Lines 135 to 145 in bcbef78

	useEffect(() => {
	const subscription = props.documents$.subscribe((next) => {
	if (next.fetchStatus !== documentState.fetchStatus) {
	if (next.result) {
	fieldCounts.current = calcFieldCounts(next.result, selectedDataView!);
	}
	setDocumentState({ ...documentState, ...next });
	}
	});
	return () => subscription.unsubscribe();
	}, [props.documents$, selectedDataView, documentState, setDocumentState]);

Here's where available fields + selected fields are sent to the field statistics by observable (Note: sending selected fields should not be necessary, since it's in our state, to send the available fields by observable is something we could reconsider, POC PR: #138238)

kibana/src/plugins/discover/public/application/main/components/sidebar/discover_sidebar_responsive.tsx

Lines 188 to 201 in bcbef78

	useEffect(
	() => {
	// For an external embeddable like the Field stats
	// it is useful to know what fields are populated in the docs fetched
	// or what fields are selected by the user
	const fieldCnts = fieldCounts.current ?? {};
	const availableFields = props.columns.length > 0 ? props.columns : Object.keys(fieldCnts);
	availableFields$.next({
	fetchStatus: FetchStatus.COMPLETE,
	fields: availableFields,
	});
	},

Here's where the grouping of the fields into (selected, popular, unpopular 😢 ) takes place:

kibana/src/plugins/discover/public/application/main/components/sidebar/discover_sidebar.tsx

Lines 153 to 160 in bcbef78

	const {
	selected: selectedFields,
	popular: popularFields,
	unpopular: unpopularFields,
	} = useMemo(
	() => groupFields(fields, columns, popularLimit, fieldCounts, fieldFilter, useNewFieldsApi),
	[fields, columns, popularLimit, fieldCounts, fieldFilter, useNewFieldsApi]
	);

This should be rewritten using in the way Lens works, which requests those fields from Elasticsearch, with the addition, that it should also work with virtual fields for our text based query initiative. With the merge of #137031 the function for getting this available fields will be migrated from server -> public.

Here's how it currently works in Lens:

kibana/x-pack/plugins/lens/public/indexpattern_datasource/datapanel.tsx

Lines 183 to 191 in 51816a0

	syncExistingFields({
	dateRange,
	setState,
	isFirstExistenceFetch: state.isFirstExistenceFetch,
	currentIndexPatternTitle: indexPatterns[currentIndexPatternId]?.title || '',
	showNoDataPopover,
	indexPatterns: indexPatternList,
	fetchJson: core.http.post,
	dslQuery,

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

Closing in favor of #135678