Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[search source] return raw response when search fails #167099

Closed
Tracked by #164350
nreese opened this issue Sep 22, 2023 · 2 comments · Fixed by #168389
Closed
Tracked by #164350

[search source] return raw response when search fails #167099

nreese opened this issue Sep 22, 2023 · 2 comments · Fixed by #168389
Assignees
@nreese
Labels
Feature:Search Querying infrastructure in Kibana impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:small Small Level of Effort Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. Team:Presentation Presentation Team for Dashboard, Input Controls, and Canvas

Comments

@nreese
Copy link
Contributor

nreese commented Sep 22, 2023 •
edited
Loading

When search fails, response does not contain raw response

Response from search source 
{
  "attributes": {
    "type": "status_exception",
    "reason": "error while executing search",
    "caused_by": {
      "type": "search_phase_execution_exception",
      "reason": "all shards failed",
      "phase": "query",
      "grouped": true,
      "failed_shards": [
        {
          "shard": 0,
          "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
          "node": "_JVoOnN5QKidGGXFJAlgpA",
          "reason": {
            "type": "query_shard_exception",
            "reason": "failed to create query: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123",
            "index_uuid": "PAa7v-dKRIyo4kv6b8dxkQ",
            "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
            "caused_by": {
              "type": "runtime_exception",
              "reason": "runtime_exception: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123"
            }
          }
        },
        {
          "shard": 0,
          "index": "remote1:kibana_sample_data_flights",
          "node": "_JVoOnN5QKidGGXFJAlgpA",
          "reason": {
            "type": "query_shard_exception",
            "reason": "failed to create query: [kibana_sample_data_flights][0] local shard failure message 123",
            "index_uuid": "MTwG4_WjSO6wG0jpDUO6DA",
            "index": "remote1:kibana_sample_data_flights",
            "caused_by": {
              "type": "runtime_exception",
              "reason": "runtime_exception: [kibana_sample_data_flights][0] local shard failure message 123"
            }
          }
        }
      ],
      "caused_by": {
        "type": "query_shard_exception",
        "reason": "failed to create query: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123",
        "index_uuid": "PAa7v-dKRIyo4kv6b8dxkQ",
        "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
        "caused_by": {
          "type": "runtime_exception",
          "reason": "runtime_exception: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123"
        }
      }
    }
  },
  "err": {
    "message": "status_exception\n\tCaused by:\n\t\tsearch_phase_execution_exception: all shards failed",
    "statusCode": 400,
    "attributes": {
      "type": "status_exception",
      "reason": "error while executing search",
      "caused_by": {
        "type": "search_phase_execution_exception",
        "reason": "all shards failed",
        "phase": "query",
        "grouped": true,
        "failed_shards": [
          {
            "shard": 0,
            "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
            "node": "_JVoOnN5QKidGGXFJAlgpA",
            "reason": {
              "type": "query_shard_exception",
              "reason": "failed to create query: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123",
              "index_uuid": "PAa7v-dKRIyo4kv6b8dxkQ",
              "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
              "caused_by": {
                "type": "runtime_exception",
                "reason": "runtime_exception: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123"
              }
            }
          },
          {
            "shard": 0,
            "index": "remote1:kibana_sample_data_flights",
            "node": "_JVoOnN5QKidGGXFJAlgpA",
            "reason": {
              "type": "query_shard_exception",
              "reason": "failed to create query: [kibana_sample_data_flights][0] local shard failure message 123",
              "index_uuid": "MTwG4_WjSO6wG0jpDUO6DA",
              "index": "remote1:kibana_sample_data_flights",
              "caused_by": {
                "type": "runtime_exception",
                "reason": "runtime_exception: [kibana_sample_data_flights][0] local shard failure message 123"
              }
            }
          }
        ],
        "caused_by": {
          "type": "query_shard_exception",
          "reason": "failed to create query: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123",
          "index_uuid": "PAa7v-dKRIyo4kv6b8dxkQ",
          "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
          "caused_by": {
            "type": "runtime_exception",
            "reason": "runtime_exception: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123"
          }
        }
      }
    }
  }
}

If you run the same request in dev tools, elasticsearch returns raw response with _shards and _clusters keys. These keys are need to populate inspector "clusters tab" to all users to diagnose failure.

response from dev tools 
{
  "id": "FmpGRWE1b2J6U0lhOE9aMkpNV3U1Nmcddk5mNEFFWUhSNS0tUzlZVndXVEh5dzoyMzcwNDA=",
  "is_partial": true,
  "is_running": false,
  "start_time_in_millis": 1695426630896,
  "expiration_time_in_millis": 1695426690896,
  "completion_time_in_millis": 1695426630900,
  "response": {
    "took": 4,
    "timed_out": false,
    "terminated_early": false,
    "_shards": {
      "total": 2,
      "successful": 2,
      "skipped": 0,
      "failed": 0
    },
    "_clusters": {
      "total": 2,
      "successful": 1,
      "skipped": 1,
      "details": {
        "(local)": {
          "status": "successful",
          "indices": "kibana_sample_data_logs,kibana_sample_data_flights",
          "took": 2,
          "timed_out": false,
          "_shards": {
            "total": 2,
            "successful": 2,
            "skipped": 0,
            "failed": 0
          }
        },
        "remote1": {
          "status": "failed",
          "indices": "kibana_sample_data*",
          "timed_out": false,
          "failures": [
            {
              "shard": -1,
              "index": null,
              "reason": {
                "type": "search_phase_execution_exception",
                "reason": "all shards failed",
                "phase": "query",
                "grouped": true,
                "failed_shards": [
                  {
                    "shard": 0,
                    "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
                    "node": "_JVoOnN5QKidGGXFJAlgpA",
                    "reason": {
                      "type": "query_shard_exception",
                      "reason": "failed to create query: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123",
                      "index_uuid": "PAa7v-dKRIyo4kv6b8dxkQ",
                      "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
                      "caused_by": {
                        "type": "runtime_exception",
                        "reason": "runtime_exception: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123"
                      }
                    }
                  },
                  {
                    "shard": 0,
                    "index": "remote1:kibana_sample_data_flights",
                    "node": "_JVoOnN5QKidGGXFJAlgpA",
                    "reason": {
                      "type": "query_shard_exception",
                      "reason": "failed to create query: [kibana_sample_data_flights][0] local shard failure message 123",
                      "index_uuid": "MTwG4_WjSO6wG0jpDUO6DA",
                      "index": "remote1:kibana_sample_data_flights",
                      "caused_by": {
                        "type": "runtime_exception",
                        "reason": "runtime_exception: [kibana_sample_data_flights][0] local shard failure message 123"
                      }
                    }
                  }
                ],
                "caused_by": {
                  "type": "query_shard_exception",
                  "reason": "failed to create query: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123",
                  "index_uuid": "PAa7v-dKRIyo4kv6b8dxkQ",
                  "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
                  "caused_by": {
                    "type": "runtime_exception",
                    "reason": "runtime_exception: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123"
                  }
                }
              }
            }
          ]
        }
      }
    },
    "hits": {
      "max_score": null,
      "hits": []
    }
  },
  "error": {
    "type": "status_exception",
    "reason": "error while executing search",
    "caused_by": {
      "type": "search_phase_execution_exception",
      "reason": "all shards failed",
      "phase": "query",
      "grouped": true,
      "failed_shards": [
        {
          "shard": 0,
          "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
          "node": "_JVoOnN5QKidGGXFJAlgpA",
          "reason": {
            "type": "query_shard_exception",
            "reason": "failed to create query: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123",
            "index_uuid": "PAa7v-dKRIyo4kv6b8dxkQ",
            "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
            "caused_by": {
              "type": "runtime_exception",
              "reason": "runtime_exception: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123"
            }
          }
        },
        {
          "shard": 0,
          "index": "remote1:kibana_sample_data_flights",
          "node": "_JVoOnN5QKidGGXFJAlgpA",
          "reason": {
            "type": "query_shard_exception",
            "reason": "failed to create query: [kibana_sample_data_flights][0] local shard failure message 123",
            "index_uuid": "MTwG4_WjSO6wG0jpDUO6DA",
            "index": "remote1:kibana_sample_data_flights",
            "caused_by": {
              "type": "runtime_exception",
              "reason": "runtime_exception: [kibana_sample_data_flights][0] local shard failure message 123"
            }
          }
        }
      ],
      "caused_by": {
        "type": "query_shard_exception",
        "reason": "failed to create query: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123",
        "index_uuid": "PAa7v-dKRIyo4kv6b8dxkQ",
        "index": "remote1:.ds-kibana_sample_data_logs-2023.09.21-000001",
        "caused_by": {
          "type": "runtime_exception",
          "reason": "runtime_exception: [.ds-kibana_sample_data_logs-2023.09.21-000001][0] local shard failure message 123"
        }
      }
    }
  }
}
@nreese nreese added Feature:Search Querying infrastructure in Kibana Team:Presentation Presentation Team for Dashboard, Input Controls, and Canvas Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. labels Sep 22, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-presentation (Team:Presentation)

@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@nreese nreese mentioned this issue Sep 22, 2023
Closed
10 tasks
@kertal kertal added impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:small Small Level of Effort labels Sep 28, 2023
@nreese nreese mentioned this issue Oct 2, 2023
Closed
@nreese nreese assigned nreese and unassigned lukasolson Oct 2, 2023
This was referenced Oct 3, 2023
Closed
Merged
nreese added a commit that referenced this issue Oct 23, 2023
@nreese @kibanamachine @MadameSheema
[search source] return rawResponse on search failure (#168389)
adf3b8b 
Closes #167099

#### Problem
`/bsearch` and `/search` APIs only return `error` key from elasticsearch
error response. This is problematic because Inspector needs
`rawResponse` to populate "Clusters and shards"

While working on this issue, I discovered another problem with how error
responses are added to inspector requestResponder. The `Error` instance
is added as `json` key. This is a little awkward since the response tab
just stringifies the contents of `json`, thus stringifing the Error
object instead of just the error body returned from API. This PR address
this problem by setting `json` to either `attributes` or `{ message }`.

#### Solution
PR updates `/bsearch` and `/search` APIs to return `{ attributes: {
error: ErrorCause, rawResponse }}` for failed responses. Solution
avoided changing KbnServerError and reportServerError since these
methods are used extensivly throughout Kibana (see
#167544 (comment) for
more details). Instead, KbnSearchError and reportSearchError are created
to report search error messages.

#### Test
1) install web logs sample data set
2) open discover
3) add filter
    ```
    {
      "error_query": {
        "indices": [
          {
            "error_type": "exception",
            "message": "local shard failure message 123",
            "name": "kibana_sample_data_logs",
            "shard_ids": [
              0
            ]
          }
        ]
      }
    }
    ```
4) Open inspector. Verify "Clusters and shards" tab is visible and
populated. Verify "Response" tab shows "error" and "rawResponse" keys.
<img width="500" alt="Screenshot 2023-10-09 at 9 29 16 AM"
src="https://github.com/elastic/kibana/assets/373691/461b0eb0-0502-4d48-a487-68025ef24d35">
<img width="500" alt="Screenshot 2023-10-09 at 9 29 06 AM"
src="https://github.com/elastic/kibana/assets/373691/9aff41eb-f771-48e3-a66d-1447689c2c6a">

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Gloria Hornero <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nreese nreese

Labels
Feature:Search Querying infrastructure in Kibana impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. loe:small Small Level of Effort Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. Team:Presentation Presentation Team for Dashboard, Input Controls, and Canvas
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[search source] return rawResponse on search failure nreese/kibana
4 participants
@nreese @kertal @lukasolson @elasticmachine