Change skip_unavailable default value to true

[quux00]

In order to improve the experience of cross-cluster search, we propose changing
the default value of the remote cluster skip_unavailable setting from false to true.

This setting causes any cross-cluster search to entirely fail when any remote cluster
with skip_unavailable=false is either unavailable (connection to it fails) or if the
search on it fails on all shards.

Setting skip_unavailable=true allows partial results from other clusters to be
returned. In that case, the search response cluster metadata will show a skipped
status, so the user can see that no data came in from that cluster. Kibana also
now leverages this metadata in the cross-cluster search responses to allow users
to see how many clusters returned data and drill down into which clusters did not
(including failure messages).

Currently, the user/admin has to specifically set the value to true in the configs, like so:

cluster:
    remote:
        remote1:
            seeds: 10.10.10.10:9300
            skip_unavailable: true

even though that is probably what search admins want in the vast majority of cases.

Setting skip_unavailable=false should be a conscious (and probably rare) choice
by an Elasticsearch admin that a particular cluster's results are so essential to a
search (or visualization in dashboard or Discover panel) that no results at all should
be shown if it cannot return any results.

[elasticsearchmachine]

Hi @quux00, I've created a changelog YAML for you. Note that since this PR is labelled >breaking, you need to update the changelog YAML to fill out the extended information sections.

[quux00]

In the process of doing this work (mostly fixing tests), I found some potential issues, both of which are pre-existing but now perhaps become more acute, so they may need to be addressed.

(1) For _search (and _async_search), when skip_unavailable=false and a cluster has an issue (such as a Security/permissions exception or IndexNotFound), that error is explicitly returned, along with an error status code such as 404 or 403. As expected, when skip_unavailable=true and the same issue occurs, an HTTP status of 200 is returned, but the error reported in the SearchResponse is NoSuchRemoteClusterException, not the IndexNotFoundException. This feels like a bug in the cluster resolution logic (es-distributed side?).

(2) The reindex API is also affected by the skip_unavailable when doing a remote reindexing operation. When skip_unavailable=false and a reindex targets an index that does not exist, then IndexNotFoundException is returned with an HTTP status error code.

However, when skip_unavailable=true it does not return an error status code, and worse it does not report any error in the response object:

{
    "took": 36,
    "timed_out": false,
    "total": 0,
    "updated": 0,
    "created": 0,
    "deleted": 0,
    "batches": 0,
    "version_conflicts": 0,
    "noops": 0,
    "retries": {
        "bulk": 0,
        "search": 0
    },
    "throttled_millis": 0,
    "requests_per_second": -1,
    "throttled_until_millis": 0,
    "failures": []
}

Should the reindex API report an issue in the failures section of this response in this case? (It likely needs to parse the search response against the remote cluster.) Or should it return an error status code? Should the reindex API have a dependency on the skip_unavailable setting?

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[quux00]

This is the only production code change. Everything else is tests and docs.

[quux00]

@elasticmachine run elasticsearch-ci/part-1

[javanna]

I left a question about yaml tests and the upgrade path, LGTM otherwise

[javanna]

Maybe obvious, but should we be explicit about the upgrade path: once you upgrade the coordinating node / cluster, where the remotes are registered, you get the new behavior? Say that the coord cluster has multiple nodes that are upgraded one by one (rolling upgrade) what would be the behavior there?

[quux00]

Good point, but that is very specific to a one-time upgrade, so should that go in the "permanent" API docs? Or should it go into some one-time doc like the changelog or deprecation announcement?

[javanna]

Normally we have migration guides for major upgrades and related changes. This is one of those changes but it goes out in a minor, I don't know if we a place for this scenario in the docs.

[quux00]

I added a blurb here about that and we'll also include in the 8.15 release notes. Thanks for flagging this.

[quux00]

@elasticmachine test this please

[quux00]

@elasticmachine run elasticsearch-ci/part-2