Enhance the opening point-in-time API

[dnhatn]

Today, we mistakenly throttle the opening point-in-time API to 1 request per node. When attempting to open a point-in-time across large clusters, it can take a significant amount of time and eventually fails due to relocated target shards or deleted target indices managed by ILM.

To address this problem, I propose the following enhancements for the opening point-in-time API:

• Increase the throttle when opening point-in-time: Increase concurrent request of opening point-in-time #96782. Opening point-in-time is a lightweight operation, doesn't perform any I/O, and is executed directly on the network threads.
• Batch opening point-in-time shard requests per node, just like we did with can_match.
• Enable minimized_round_trips when opening point-in-time. Either option 2 or option 3 should be fine. It seems both require handling backward compatibility (wire or security).
• Add the option to allow partial results when opening point-in-time [Open PIT API incorrectly suggests using allow_partial_search_results if a shard is unavailable #103165]
• Support skip_unavailable when open PIT is used against remote clusters [Support skip_unavailable when opening a CCS PIT #103166]
• Add the ability to accept a filter when opening a point-in-time. This would reduce the number of target shards and the PIT id (Add support for index_filter to open pit #102388).
• Consider exposing reader_id in the cluster state for searchable snapshots to avoid reaching out to the frozen/cold nodes when opening point-in-time.
• support PIT when calling search shards against remote clusters, which currently does not execute the can match phase and is a reason why we cannot remove the legacy cluster search shards API: https://github.com/elastic/elasticsearch/blob/main/server/src/main/java/org/elasticsearch/action/search/TransportSearchAction.java#L712

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[rseldner]

looks like this item can be checked off, yes?

Add the option to allow partial results when opening point-in-time [https://github.com//issues/103165]

PR: #111516