Cross cluster search causes errors when is_partial: true in the response

[thomasneirynck]

Kibana version: 8.10.0

Elasticsearch version: 8.10.0

Steps to reproduce:

1. Spin up an ES instance (this will be the remote)
2. Ingest the following documents:

POST tmp-00001/_doc
{
  "foo": "bar",
  "bar": 1
}

POST tmp-00002/_doc
{
  "foo": "baz"
}

3. Spin up a separate Kibana/ES instance
4. Under Stack Management → Remote Clusters, add the cluster from step 1
5. Create a Data View that corresponds with the remote indices (*:tmp-*)
6. Go to Discover and select the data view (should work fine)
7. Add a filter that uses a script that causes errors in one of the remote shards (using "Edit as Query DSL"):

{"script": {"script": "doc['bar'].value < 10"}}

You will get a "Received partial response" error message with no results.

Expected behavior:

An error message with the shard failures, and seeing the one document that matches the filter.

Any additional context:

Similar configurations also fail in the Security app

Looking into the exact request/response, it seems as though there have been changes in the response between 8.9 and 8.10. To illustrate this, in the above setup, you can run the following query:

POST *:tmp-*/_async_search?ccs_minimize_roundtrips=true
{
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "script": {
            "script": "doc['bar'].value < 10"
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

In 8.10.0, ES will return "is_partial": true, whereas in 8.9.2, ES will return "is_partial": false.

In Kibana, we are using these flags to determine whether or not the response is an error:

kibana/src/plugins/data/common/search/utils.ts

Line 17 in c95acfc

return !response || !response.rawResponse || (!response.isRunning && !!response.isPartial);

This logic is incorrect and relies on a bug in ES that returns "is_partial": false whenever "is_running" is also false. For context, see #164893 and elastic/elasticsearch#98725.

The main problem now is that ES sometimes correctly reports "is_partial": true when some remote shards fail while others succeed, but still incorrectly reports "is_partial": false for non-CCS setups. (See elastic/elasticsearch#97731 and elastic/elasticsearch#98913.)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[drewdaemon]

This is definitely a problem for visualizations as well. I followed the STR and got errors where I should have seen warnings:

Lens editor

Dashboard

Aggs-based editor

[mbudge]

This bug severely impacted both security and IT users. Going forward we're not going to upgrade to a new minor version of elastic until at least 2 maintenance releases are available. The reason for this is this bug won't be fixed until the second maintenance release, which makes me think second maintenance releases are more stable. Please review your testing processes as CCS should have been tested in Kibana before changes went GA.

[sophiec20]

Yes, we are reviewing our testing for CCS and also looking to improve how we handle changes to elasticsearch API responses between versions.

[kertal]

@lukasolson Guess we can close this now, right?

[kertal]

@lukasolson let's resolve it with #166667