[ML] Delete annotation method assumes annotation is in current write index

[droberts195]

The deleteAnnotation function in ML assumes that the annotation to be deleted is in the index that the annotations write alias currently points to:

kibana/x-pack/plugins/ml/server/models/annotation_service/annotation.ts

Lines 409 to 418 in a251dbf

	async function deleteAnnotation(id: string) {
	const params: DeleteParams = {
	index: ML_ANNOTATIONS_INDEX_ALIAS_WRITE,
	id,
	refresh: 'wait_for',
	};
	const { body } = await asInternalUser.delete(params);
	return body;
	}

This assumption is incorrect. The annotation to be deleted could be in any one of the indices that the annotations read alias points to.

There are two possible ways to fix this:

1. When reading annotations into memory, each annotation object remembers which index it was stored it. Then, if it is deleted, that index is passed to the delete function.
2. The delete function does a delete-by-query on the read alias using an IDs search with the ID of the annotation to be deleted. This will generally find 1 document in one of the indices that the read alias points to, and 0 documents in all the others.

I'll leave it up to the UI team to decide which is easiest to implement.

This needs to be fixed for 7.16, because then in 8.0 we'll have the chance to actually have multiple annotations indices behind the aliases. (We have got away with this bug up to now because there's only been one index that both the read and write aliases point at. But the whole point of having separate read and write aliases was to give the flexibility to have multiple backing indices.)