Saved objects (dashboard type) delete time is inconsistent

[dmlemeshko]

Starting April 25 we noticed a change in saved object delete_time metric for dashboard_listing_page journey:

Afaik this metric represents how much time it takes to delete saved object (dashboard)

kibana/src/plugins/dashboard/public/dashboard_listing/dashboard_listing.tsx

Lines 162 to 192 in f664e49

	const deleteItems = useCallback(
	async (dashboardsToDelete: Array<{ id: string }>) => {
	try {
	const deleteStartTime = window.performance.now();
	await Promise.all(
	dashboardsToDelete.map(({ id }) => {
	dashboardSessionStorage.clearState(id);
	return savedObjectsClient.delete(DASHBOARD_SAVED_OBJECT_TYPE, id);
	})
	);
	const deleteDuration = window.performance.now() - deleteStartTime;
	reportPerformanceMetricEvent(pluginServices.getServices().analytics, {
	eventName: SAVED_OBJECT_DELETE_TIME,
	duration: deleteDuration,
	meta: {
	saved_object_type: DASHBOARD_SAVED_OBJECT_TYPE,
	total: dashboardsToDelete.length,
	},
	});
	} catch (error) {
	toasts.addError(error, {
	title: dashboardListingErrorStrings.getErrorDeletingDashboardToast(),
	});
	}
	setUnsavedDashboardIds(dashboardSessionStorage.getDashboardIdsWithUnsavedChanges());
	},
	[savedObjectsClient, dashboardSessionStorage, toasts]
	);

Time is matching #154888 PR that splits .kibana saved objects index into multiple indices

Before the change delete_time was pretty constant taking ~600ms. After a change delete_time is not consistent taking 150-1000 ms

Scenario (delete step) is pretty simple:

kibana/x-pack/performance/journeys/dashboard_listing_page.ts

Lines 28 to 33 in f664e49

	.step('Delete dashboard', async ({ page }) => {
	await page.click('[data-test-subj="checkboxSelectRow-edf84fe0-e1a0-11e7-b6d5-4dc382ef7f5b"]');
	await page.click('[data-test-subj="deleteSelectedItems"]');
	await page.click('[data-test-subj="confirmModalConfirmButton"]');
	await page.waitForSelector(`[data-test-subj="table-is-ready"]`);
	})

You can more information on performance dashboard

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[dmlemeshko]

Closing since metric remains within 1s limit. We adjusted alters to not track a step change, but track upper limits for both delete (1100ms) and post (600ms) operations.