Add a tie_breaker field to all saved objects

[rudolf]

In order to efficiently search across > 10k saved objects using the recommended search_after API without using a PIT, all saved objects documents needs to have a unique tie breaker field.

The tie_breaker field needs to be unique across all documents but there is no unique constraint in Elasticsearch, so simply copying the document _id into the tie-breaker field provides the best uniqueness guarantee. This has to be the _id field on the SavedObjectsRawDoc (which includes the type and sometimes namespace) not the id field on the serialized SavedObjectDoc (since this is not unique when users copy documents to other spaces)

Since the existing saved object migration framework works on serialized SavedObjectDoc documents we would either need to use the generateRawId method from the SavedObjectsSerializer or implement this migration in another layer. This could for instance be a scripted reindex which migrates all existing documents and then the SavedObjectsSerializer could populate this field for any documents that are added from the SavedObjectsClient/Repository.

If we want to implement this using existing migrations we would have to:

• add support for 'core' migrations
• add a 'core' migration to populate the tie breaker field on all our existing objects
• if we use existing migrationVersion field of all saved object types that gets exposed to clients, migrations will automatically run on any documents before we write them so this will ensure that new documents also have a tie breaker field

Blocks #77961 #62002

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[jportner]

Note: in #80945 we add support for defining additional transforms based on a common coreMigrationVersion saved object field that is tied to the current Kibana version. After that is merged, we can define a new 'core' transform type and a migration that would add this tie_breaker field to all existing and new saved objects.

[lukeelmers]

Once elastic/elasticsearch#68833 lands, a _shard_doc tiebreaker will automatically applied internally to any search using a PIT.

So if we don't have a need in Kibana for a SO tiebreaker other than to use with search_after and pit, we can probably close this issue.

[lukeelmers]

elastic/elasticsearch#68833 has landed and I've opened #92295 to clean up our manual usage of the _shard_doc tiebreaker.

This means that moving forward, a tiebreaker is not necessary for search requests to ES that are using a PIT, as it will automatically be applied behind the scenes.

Hearing no opposition to my earlier comment, I'll plan to close this issue once it is confirmed that it is no longer blocking #62002

[rudolf]

+1 There could be valid use cases to have consistent sort ordering without using a PIT, but if we're not aware of anyone being blocked on it, we should rather close it for now and reopen once there is a need.

[lukeelmers]

There hasn't been any further activity on #62002, so I will go ahead and close this for now. We can reopen if there turns out to be a need for it after all.