[Discover] Field List - refactor sections to align with Lens implementation

[kertal]

Part 1 #142758

• Extract Lens components to UnifiedFieldList plugin
• Create a new hook for fetching data
• Integrate these components back to Lens page
• Refactor field list implementation for text based queries in Lens (separate implementation with classes)

Part 2:

• Integrate these components to Discover. They will replace existing list sections and add Meta/Empty sections
• Remove "Hide empty fields" filter from Discover
• [Discover] Decouple Field Statistics tab from Discover fields sidebar by refactoring how availableFields are used #140367
• Clean up old Discover code
• Support field list for text based queries in Discover

More context

Currently in Discover, we calculate the available fields by the data returned by Elasticsearch.

To show all fields users need to change the Hide empty fields filter

Here's where the field statistics are calculated:

kibana/src/plugins/discover/public/application/main/components/sidebar/discover_sidebar_responsive.tsx

Lines 135 to 145 in bcbef78

	useEffect(() => {
	const subscription = props.documents$.subscribe((next) => {
	if (next.fetchStatus !== documentState.fetchStatus) {
	if (next.result) {
	fieldCounts.current = calcFieldCounts(next.result, selectedDataView!);
	}
	setDocumentState({ ...documentState, ...next });
	}
	});
	return () => subscription.unsubscribe();
	}, [props.documents$, selectedDataView, documentState, setDocumentState]);

Here's where available fields + selected fields are sent to the field statistics by observable (Note: sending selected fields should not be necessary, since it's in our state, to send the available fields by observable is something we could reconsider, POC PR: #138238)

kibana/src/plugins/discover/public/application/main/components/sidebar/discover_sidebar_responsive.tsx

Lines 188 to 201 in bcbef78

	useEffect(
	() => {
	// For an external embeddable like the Field stats
	// it is useful to know what fields are populated in the docs fetched
	// or what fields are selected by the user
	const fieldCnts = fieldCounts.current ?? {};
	const availableFields = props.columns.length > 0 ? props.columns : Object.keys(fieldCnts);
	availableFields$.next({
	fetchStatus: FetchStatus.COMPLETE,
	fields: availableFields,
	});
	},

Here's where the grouping of the fields into (selected, popular, unpopular 😢 ) takes place:

kibana/src/plugins/discover/public/application/main/components/sidebar/discover_sidebar.tsx

Lines 153 to 160 in bcbef78

	const {
	selected: selectedFields,
	popular: popularFields,
	unpopular: unpopularFields,
	} = useMemo(
	() => groupFields(fields, columns, popularLimit, fieldCounts, fieldFilter, useNewFieldsApi),
	[fields, columns, popularLimit, fieldCounts, fieldFilter, useNewFieldsApi]
	);

This should be rewritten using in the way Lens works, which requests those fields from Elasticsearch, with the addition, that it should also work with virtual fields for our text based query initiative. With the merge of #137031 the function for getting this available fields will be migrated from server -> public.

Here's how it currently works in Lens:

kibana/x-pack/plugins/lens/public/indexpattern_datasource/datapanel.tsx

Lines 183 to 191 in 51816a0

	syncExistingFields({
	dateRange,
	setState,
	isFirstExistenceFetch: state.isFirstExistenceFetch,
	currentIndexPatternTitle: indexPatterns[currentIndexPatternId]?.title || '',
	showNoDataPopover,
	indexPatterns: indexPatternList,
	fetchJson: core.http.post,
	dslQuery,

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[jughosta]

Dima is working on API refactoring #139453

[jughosta]

Reposting the note from the meta issue here:

Currently we use the sidebar to submit existing fields to field statistics this should be improved 2 approaches: centralize field existence to the saved search data fetching OR decentralize & decouple (e.h. using the same hook)

kibana/src/plugins/discover/public/application/main/components/sidebar/discover_sidebar_responsive.tsx

Lines 188 to 211 in a8ebf5a

	useEffect(
	() => {
	// For an external embeddable like the Field stats
	// it is useful to know what fields are populated in the docs fetched
	// or what fields are selected by the user
	const fieldCnts = fieldCounts.current ?? {};
	const availableFields = props.columns.length > 0 ? props.columns : Object.keys(fieldCnts);
	availableFields$.next({
	fetchStatus: FetchStatus.COMPLETE,
	fields: availableFields,
	});
	},
	// Using columns.length here instead of columns to avoid array reference changing
	// eslint-disable-next-line react-hooks/exhaustive-deps
	[
	selectedDataView,
	availableFields$,
	fieldCounts.current,
	documentState.result,
	props.columns.length,
	]
	);

[kertal]

@jughosta thx, I think one of the next steps could be, create a hook in Discover that could fetch the existing fields, and use it in field statistics, remove the logic of the observable that we populate in the sidebar to use in the Field Statistic tab. feels this would be a good start for the field statistics work later on (that we should discuss separately)