Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cross-cluster search with minimize_roundtrips=true reports clusters with no shards to search as failed or skipped #100350

Closed
quux00 opened this issue Oct 5, 2023 · 1 comment
Closed

Cross-cluster search with minimize_roundtrips=true reports clusters with no shards to search as failed or skipped #100350

quux00 opened this issue Oct 5, 2023 · 1 comment
Assignees
@quux00
Labels
>bug :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team

Comments

@quux00
Copy link
Contributor

quux00 commented Oct 5, 2023

Elasticsearch Version

8.10

Installed Plugins

No response

Java Version

bundled

OS Version

Darwin michaels-mbp.lan 22.6.0 Darwin Kernel Version 22.6.0: Fri Sep 15 13:41:28 PDT 2023; root:xnu-8796.141.3.700.8~1/RELEASE_ARM64_T6000 arm64

Problem Description

Cross cluster searches done with ccs_minimize_roundtrips=true (the default in Kibana as of version 8.10) inappropriately marks searches on remote clusters as "failed" or "skipped" when there are no shards to search (e.g., when that index is not present on the remote cluster). Those clusters should instead be reporting as successful. See "Steps to Reproduce" section for an example of correct vs. incorrect behavior.

This bug causes: #100311

Steps to Reproduce

Do a cross-cluster where one or more of the remote clusters lacks an index and query with ccs_minimize_roundtrips=true.

Example:

POST *,remote*:blog*/_async_search?ccs_minimize_roundtrips=true
{
  "size": 1,
  "query": {
    "match_all": {}
  }
}

Result:

    "_clusters": {
      "total": 3,
      "successful": 1,
      "skipped": 1,
      "running": 0,
      "partial": 0,
      "failed": 1,
      "details": {
        "(local)": {
          "status": "successful",
          "indices": "*",
          "took": 24,
          "timed_out": false,
          "_shards": {
            "total": 2,
            "successful": 2,
            "skipped": 0,
            "failed": 0
          }
        },
        "remote2": {
          "status": "skipped",
          "indices": "blog*",
          "took": 2,
          "timed_out": false,
          "_shards": {
            "total": 0,
            "successful": 0,
            "skipped": 0,
            "failed": 0
          }
        },
        "remote1": {
          "status": "failed",
          "indices": "blog*",
          "took": 0,
          "timed_out": false,
          "_shards": {
            "total": 0,
            "successful": 0,
            "skipped": 0,
            "failed": 0
          }
        }
      }
    },

The correct response (which you get with ccs_minimize_roundtrips=false) should be:

   "_clusters": {
      "total": 3,
      "successful": 3,
      "skipped": 0,
      "running": 0,
      "partial": 0,
      "failed": 0,
      "details": {
        "(local)": {
          "status": "successful",
          "indices": "*",
          "took": 21,
          "timed_out": false,
          "_shards": {
            "total": 2,
            "successful": 2,
            "skipped": 0,
            "failed": 0
          }
        },
        "remote2": {
          "status": "successful",
          "indices": "blog*",
          "took": 16,
          "timed_out": false,
          "_shards": {
            "total": 0,
            "successful": 0,
            "skipped": 0,
            "failed": 0
          }
        },
        "remote1": {
          "status": "successful",
          "indices": "blog*",
          "took": 16,
          "timed_out": false,
          "_shards": {
            "total": 0,
            "successful": 0,
            "skipped": 0,
            "failed": 0
          }
        }
      }
    },

Logs (if relevant)

No response
@quux00 quux00 added >bug needs:triage Requires assignment of a team area label :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team and removed needs:triage Requires assignment of a team area label labels Oct 5, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@quux00 quux00 self-assigned this Oct 5, 2023
quux00 added a commit to quux00/elasticsearch that referenced this issue Oct 5, 2023
@quux00
Search of remote clusters with no shards results in successful status.
b6270d7 
ccs_minimize_roundtrips=true was not checking if no shards were present to search
before setting the cluster search status. If number of shards is zero, then cluster
search status should be zero. This aligns ccs_minimize_roundtrips=true with the
false setting and how it worked in earlier versions.

Fixes bug elastic#100350
@quux00 quux00 mentioned this issue Oct 5, 2023
Merged
quux00 added a commit that referenced this issue Oct 6, 2023
@quux00
Search of remote clusters with no shards results in successful status. (
a708501 
#100354)

ccs_minimize_roundtrips=true was not checking if no shards were present to search
before setting the cluster search status. If the number of shards is zero, then cluster
search status should be 'successful'. This aligns ccs_minimize_roundtrips=true with the
false setting and how it worked in earlier versions.

Fixes bug #100350
quux00 added a commit to quux00/elasticsearch that referenced this issue Oct 6, 2023
@quux00
Search of remote clusters with no shards results in successful status. (
b34574d 
elastic#100354)

ccs_minimize_roundtrips=true was not checking if no shards were present to search
before setting the cluster search status. If the number of shards is zero, then cluster
search status should be 'successful'. This aligns ccs_minimize_roundtrips=true with the
false setting and how it worked in earlier versions.

Fixes bug elastic#100350
quux00 added a commit to quux00/elasticsearch that referenced this issue Oct 6, 2023
@quux00
Search of remote clusters with no shards results in successful status. (
3268cd2 
elastic#100354)

ccs_minimize_roundtrips=true was not checking if no shards were present to search
before setting the cluster search status. If the number of shards is zero, then cluster
search status should be 'successful'. This aligns ccs_minimize_roundtrips=true with the
false setting and how it worked in earlier versions.

Fixes bug elastic#100350
elasticsearchmachine pushed a commit that referenced this issue Oct 6, 2023
@quux00
Search of remote clusters with no shards results in successful status. (
35ef8a1 
#100354) (#100442)

ccs_minimize_roundtrips=true was not checking if no shards were present to search
before setting the cluster search status. If the number of shards is zero, then cluster
search status should be 'successful'. This aligns ccs_minimize_roundtrips=true with the
false setting and how it worked in earlier versions.

Fixes bug #100350
quux00 added a commit to quux00/elasticsearch that referenced this issue Oct 6, 2023
@quux00
Search of remote clusters with no shards results in successful status. (
5a587b7 
elastic#100354)

ccs_minimize_roundtrips=true was not checking if no shards were present to search
before setting the cluster search status. If the number of shards is zero, then cluster
search status should be 'successful'. This aligns ccs_minimize_roundtrips=true with the
false setting and how it worked in earlier versions.

Fixes bug elastic#100350
quux00 added a commit that referenced this issue Oct 6, 2023
@quux00
Search of remote clusters with no shards results in successful status. (
4210a29 
#100354) (#100458)

ccs_minimize_roundtrips=true was not checking if no shards were present to search
before setting the cluster search status. If the number of shards is zero, then cluster
search status should be 'successful'. This aligns ccs_minimize_roundtrips=true with the
false setting and how it worked in earlier versions.

Backported from #100354 

Fixes bug #100350
@quux00 quux00 closed this as completed Oct 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@quux00 quux00

Labels
>bug :Search/Search Search-related issues that do not fall into other categories Team:Search Meta label for search team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@quux00 @elasticsearchmachine