[Discover] Default Columns do not always apply with switching Data Views

[kertal]

Kibana version:
main, 8.11, 8.10, ....

Describe the bug:
When switching Data Views in Discover, the Advanced Setting "Default Columns" does not always apply.

Steps to reproduce:

1. Add Kibana Sample Data Logs and Kibana Sample eCommerce Data of our Sample data
2. In Advanced settings / Discover / Default Columns enter timestamp, message, agent and save this setting
3. Navigate to Discover
4. Select Kibana Sample Data Logs of the data view picker
5. Now you should be able to see the configured columns
6. Select Kibana Sample eCommerce Data of the data view picker
7. Now you should be able to see timestamp, Document as columns, this is expected, unless you configured discover:modifyColumnsOnSwitch to Off, which prevents modifying the columns on switching
8. Select Kibana Sample Data Logs of the data view picker, again
9. Now the columns are still timestamp, Document

Expected behavior:
When switching back to Kibana Sample Data Logs, the configured Default Columns should be displayed

** Recording **

Kapture.2023-10-11.at.12.27.33.mp4

Context
Here's the code responsible for the next state when switching data views:

kibana/src/plugins/discover/public/application/main/utils/get_switch_data_view_app_state.ts

Lines 17 to 65 in 1a213bd

	export function getDataViewAppState(
	currentDataView: DataView,
	nextDataView: DataView,
	currentColumns: string[],
	currentSort: SortOrder[],
	modifyColumns: boolean = true,
	sortDirection: string = 'desc',
	query?: Query | AggregateQuery
	) {
	const nextColumns = modifyColumns
	? currentColumns.filter(
	(column) =>
	nextDataView.fields.getByName(column) || !currentDataView.fields.getByName(column)
	)
	: currentColumns;
	let columns = nextColumns.length ? nextColumns : [];
	if (query && isOfAggregateQueryType(query)) {
	columns = [];
	}
	// when switching from an data view with timeField to an data view without timeField
	// filter out sorting by timeField in case it is set. data views without timeField don't
	// prepend this field in the table, so in legacy grid you would need to add this column to
	// remove sorting
	let nextSort = getSortArray(currentSort, nextDataView).filter((value) => {
	return nextDataView.timeFieldName || value[0] !== currentDataView.timeFieldName;
	});
	if (nextDataView.isTimeBased() && !nextSort.length) {
	// set default sorting if it was not set
	nextSort = [[nextDataView.timeFieldName, sortDirection]];
	} else if (
	nextDataView.isTimeBased() &&
	currentDataView.isTimeBased() &&
	nextDataView.timeFieldName !== currentDataView.timeFieldName
	) {
	// switch time fields
	nextSort = nextSort.map((cur) =>
	cur[0] === currentDataView.timeFieldName ? [nextDataView.timeFieldName, cur[1]] : cur
	);
	}
	return {
	index: nextDataView.id,
	columns,
	sort: nextSort,
	};
	}

Additional info
Reported by @sophiec20

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)