Need performant method of determining whether there are indices #112307
Labels
:Data Management/Indices APIs
APIs to create and manage indices and templates
>enhancement
Team:Data Management
Meta label for data/management team
Comments
mattkime
added
>enhancement
needs:triage
davismcphee
added a commit
to elastic/kibana
that referenced
this issue
Sep 4, 2024
… for improved performance (#191566) ## Summary The `resolve/cluster` api is MUCH more efficient for determining whether there are user created indices than the `resolve/index` api. The `resolve/index` api returns the FULL list of indices which can be very large. Unfortunately the `resolve/cluster` api isn't available on serverless so we rely on the existing `hasESData` behavior when its not available. Closes #190554 Created elastic/elasticsearch#112307 in hopes of getting an api thats performant in serverless and classic environments. Additional detail - `logs-enterprise_search.api-default` and `logs-enterprise_search.audit-default` should be ignored for the purposes of user created data. --- Testing - verify the loading data flows display as appropriate for discover and data view management. Create and delete indices. ## Release notes In deployments with thousands of indices and index aliases, browser calls to `/internal/index-pattern-management/resolve_index` can be very slow (more than 10s). Its been replaced with `/internal/data_views/has_es_data` which is much faster (<1s). --------- Co-authored-by: Davis McPhee <[email protected]>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Sep 4, 2024
… for improved performance (elastic#191566) ## Summary The `resolve/cluster` api is MUCH more efficient for determining whether there are user created indices than the `resolve/index` api. The `resolve/index` api returns the FULL list of indices which can be very large. Unfortunately the `resolve/cluster` api isn't available on serverless so we rely on the existing `hasESData` behavior when its not available. Closes elastic#190554 Created elastic/elasticsearch#112307 in hopes of getting an api thats performant in serverless and classic environments. Additional detail - `logs-enterprise_search.api-default` and `logs-enterprise_search.audit-default` should be ignored for the purposes of user created data. --- Testing - verify the loading data flows display as appropriate for discover and data view management. Create and delete indices. ## Release notes In deployments with thousands of indices and index aliases, browser calls to `/internal/index-pattern-management/resolve_index` can be very slow (more than 10s). Its been replaced with `/internal/data_views/has_es_data` which is much faster (<1s). --------- Co-authored-by: Davis McPhee <[email protected]> (cherry picked from commit 86cfcab)
kibanamachine
added a commit
to elastic/kibana
that referenced
this issue
Sep 5, 2024
…e index for improved performance (#191566) (#192143) # Backport This will backport the following commits from `main` to `8.15`: - [[data views / hasData] Check resolve cluster instead of resolve index for improved performance (#191566)](#191566) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Matthew Kime","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-09-04T23:36:35Z","message":"[data views / hasData] Check resolve cluster instead of resolve index for improved performance (#191566)\n\n## Summary\r\n\r\nThe `resolve/cluster` api is MUCH more efficient for determining whether\r\nthere are user created indices than the `resolve/index` api. The\r\n`resolve/index` api returns the FULL list of indices which can be very\r\nlarge.\r\n\r\nUnfortunately the `resolve/cluster` api isn't available on serverless so\r\nwe rely on the existing `hasESData` behavior when its not available.\r\n\r\nCloses https://github.com/elastic/kibana/issues/190554\r\n\r\nCreated elastic/elasticsearch#112307 in hopes\r\nof getting an api thats performant in serverless and classic\r\nenvironments. Additional detail - `logs-enterprise_search.api-default`\r\nand `logs-enterprise_search.audit-default` should be ignored for the\r\npurposes of user created data.\r\n\r\n---\r\n\r\nTesting - verify the loading data flows display as appropriate for\r\ndiscover and data view management. Create and delete indices.\r\n\r\n## Release notes\r\n\r\nIn deployments with thousands of indices and index aliases, browser\r\ncalls to `/internal/index-pattern-management/resolve_index` can be very\r\nslow (more than 10s). Its been replaced with\r\n`/internal/data_views/has_es_data` which is much faster (<1s).\r\n\r\n---------\r\n\r\nCo-authored-by: Davis McPhee <[email protected]>","sha":"86cfcab8770a5b31e73c179baad53ee382c837f4","branchLabelMapping":{"^v8.16.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Feature:Data Views","Team:DataDiscovery","backport:prev-minor","ci:project-deploy-security","v8.16.0"],"title":"[data views / hasData] Check resolve cluster instead of resolve index for improved performance","number":191566,"url":"https://github.com/elastic/kibana/pull/191566","mergeCommit":{"message":"[data views / hasData] Check resolve cluster instead of resolve index for improved performance (#191566)\n\n## Summary\r\n\r\nThe `resolve/cluster` api is MUCH more efficient for determining whether\r\nthere are user created indices than the `resolve/index` api. The\r\n`resolve/index` api returns the FULL list of indices which can be very\r\nlarge.\r\n\r\nUnfortunately the `resolve/cluster` api isn't available on serverless so\r\nwe rely on the existing `hasESData` behavior when its not available.\r\n\r\nCloses https://github.com/elastic/kibana/issues/190554\r\n\r\nCreated elastic/elasticsearch#112307 in hopes\r\nof getting an api thats performant in serverless and classic\r\nenvironments. Additional detail - `logs-enterprise_search.api-default`\r\nand `logs-enterprise_search.audit-default` should be ignored for the\r\npurposes of user created data.\r\n\r\n---\r\n\r\nTesting - verify the loading data flows display as appropriate for\r\ndiscover and data view management. Create and delete indices.\r\n\r\n## Release notes\r\n\r\nIn deployments with thousands of indices and index aliases, browser\r\ncalls to `/internal/index-pattern-management/resolve_index` can be very\r\nslow (more than 10s). Its been replaced with\r\n`/internal/data_views/has_es_data` which is much faster (<1s).\r\n\r\n---------\r\n\r\nCo-authored-by: Davis McPhee <[email protected]>","sha":"86cfcab8770a5b31e73c179baad53ee382c837f4"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.16.0","branchLabelMappingKey":"^v8.16.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/191566","number":191566,"mergeCommit":{"message":"[data views / hasData] Check resolve cluster instead of resolve index for improved performance (#191566)\n\n## Summary\r\n\r\nThe `resolve/cluster` api is MUCH more efficient for determining whether\r\nthere are user created indices than the `resolve/index` api. The\r\n`resolve/index` api returns the FULL list of indices which can be very\r\nlarge.\r\n\r\nUnfortunately the `resolve/cluster` api isn't available on serverless so\r\nwe rely on the existing `hasESData` behavior when its not available.\r\n\r\nCloses https://github.com/elastic/kibana/issues/190554\r\n\r\nCreated elastic/elasticsearch#112307 in hopes\r\nof getting an api thats performant in serverless and classic\r\nenvironments. Additional detail - `logs-enterprise_search.api-default`\r\nand `logs-enterprise_search.audit-default` should be ignored for the\r\npurposes of user created data.\r\n\r\n---\r\n\r\nTesting - verify the loading data flows display as appropriate for\r\ndiscover and data view management. Create and delete indices.\r\n\r\n## Release notes\r\n\r\nIn deployments with thousands of indices and index aliases, browser\r\ncalls to `/internal/index-pattern-management/resolve_index` can be very\r\nslow (more than 10s). Its been replaced with\r\n`/internal/data_views/has_es_data` which is much faster (<1s).\r\n\r\n---------\r\n\r\nCo-authored-by: Davis McPhee <[email protected]>","sha":"86cfcab8770a5b31e73c179baad53ee382c837f4"}}]}] BACKPORT--> Co-authored-by: Matthew Kime <[email protected]>
javanna
added
:Data Management/Indices APIs
elasticsearchmachine
added
the
Team:Data Management
|
Pinging @elastic/es-data-management (Team:Data Management) |
|
I labeled this data management because that's the team that cares for indices level API, like resolve index. Turns out that this issue has been overcome by using resolve cluster instead against Elasticsearch. Such API is built to resolve clusters though, and I wonder whether there is a different way that I am not aware of. Or perhaps if we need a specific API for this purpose. It seems like a bug that none of our API serve valid purpose besides one that was built for an entirely different scenario :) |
davismcphee
added a commit
to elastic/kibana
that referenced
this issue
Nov 20, 2024
…a to hang (#200476) ## Summary This PR mitigates an issue where the `has_es_data` check can hang when some remote clusters are unresponsive, leaving users stuck in a loading state in some apps (e.g. Discover and Dashboard) until the request times out. There are two main changes that help mitigate this issue: - The `resolve/cluster` request in the `has_es_data` endpoint has been split into two requests -- one for local data first, then another for remote data second. In cases where remote clusters are unresponsive but there is data available in the local cluster, the remote check is never performed and the check completes quickly. This likely resolves the majority of cases and is also likely faster in general than checking both local and remote clusters in a single request. - In cases where there is no local data and the remote `resolve/cluster` request hangs, a new `data_views.hasEsDataTimeout` config has been added to `kibana.yml` (defaults to 5 seconds) to abort the request after a short delay. This scenario is handled in the front end by displaying an error toast to the user informing them of the issue, and assuming there is data available to avoid blocking them. When this occurs, a warning is also logged to the Kibana server logs.  Fixes #200280. ### Notes - Modifying the existing version of the `has_es_data` endpoint in this way should be backward compatible since the behaviour should remain unchanged from before when the client and server versions don't match (please validate if this seems accurate during review). - For a long term fix, the ES team is investigating the issue with `resolve/cluster` and will aim to have it behave like `resolve/index`, which fails quickly when remote clusters are unresponsive. They may also implement other mitigations like a configurable timeout in ES: elastic/elasticsearch#114020. The purpose of this PR is to provide an immediate solution in Kibana that mitigates the issue as much as possible. - If ES ends up providing another performant method for checking if indices exist instead of `resolve/cluster`, Kibana should migrate to that. More details in elastic/elasticsearch#112307. ### Testing notes To reproduce the issue locally, follow these steps: - Follow [these instructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756) to set up a local CCS environment. - Stop the remote cluster process. - Use Netcat on the remote cluster port to listen to requests but not respond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive cluster. See elastic/elasticsearch#32678 for more context. - Navigate to Discover and observe that the `has_es_data` request hangs. When testing in this PR branch, the request will only wait for 5 seconds before assuming data exists and displaying a toast. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_node:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]>
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Nov 20, 2024
…a to hang (elastic#200476) ## Summary This PR mitigates an issue where the `has_es_data` check can hang when some remote clusters are unresponsive, leaving users stuck in a loading state in some apps (e.g. Discover and Dashboard) until the request times out. There are two main changes that help mitigate this issue: - The `resolve/cluster` request in the `has_es_data` endpoint has been split into two requests -- one for local data first, then another for remote data second. In cases where remote clusters are unresponsive but there is data available in the local cluster, the remote check is never performed and the check completes quickly. This likely resolves the majority of cases and is also likely faster in general than checking both local and remote clusters in a single request. - In cases where there is no local data and the remote `resolve/cluster` request hangs, a new `data_views.hasEsDataTimeout` config has been added to `kibana.yml` (defaults to 5 seconds) to abort the request after a short delay. This scenario is handled in the front end by displaying an error toast to the user informing them of the issue, and assuming there is data available to avoid blocking them. When this occurs, a warning is also logged to the Kibana server logs.  Fixes elastic#200280. ### Notes - Modifying the existing version of the `has_es_data` endpoint in this way should be backward compatible since the behaviour should remain unchanged from before when the client and server versions don't match (please validate if this seems accurate during review). - For a long term fix, the ES team is investigating the issue with `resolve/cluster` and will aim to have it behave like `resolve/index`, which fails quickly when remote clusters are unresponsive. They may also implement other mitigations like a configurable timeout in ES: elastic/elasticsearch#114020. The purpose of this PR is to provide an immediate solution in Kibana that mitigates the issue as much as possible. - If ES ends up providing another performant method for checking if indices exist instead of `resolve/cluster`, Kibana should migrate to that. More details in elastic/elasticsearch#112307. ### Testing notes To reproduce the issue locally, follow these steps: - Follow [these instructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756) to set up a local CCS environment. - Stop the remote cluster process. - Use Netcat on the remote cluster port to listen to requests but not respond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive cluster. See elastic/elasticsearch#32678 for more context. - Navigate to Discover and observe that the `has_es_data` request hangs. When testing in this PR branch, the request will only wait for 5 seconds before assuming data exists and displaying a toast. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_node:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit 96fd4b6)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Nov 20, 2024
…a to hang (elastic#200476) ## Summary This PR mitigates an issue where the `has_es_data` check can hang when some remote clusters are unresponsive, leaving users stuck in a loading state in some apps (e.g. Discover and Dashboard) until the request times out. There are two main changes that help mitigate this issue: - The `resolve/cluster` request in the `has_es_data` endpoint has been split into two requests -- one for local data first, then another for remote data second. In cases where remote clusters are unresponsive but there is data available in the local cluster, the remote check is never performed and the check completes quickly. This likely resolves the majority of cases and is also likely faster in general than checking both local and remote clusters in a single request. - In cases where there is no local data and the remote `resolve/cluster` request hangs, a new `data_views.hasEsDataTimeout` config has been added to `kibana.yml` (defaults to 5 seconds) to abort the request after a short delay. This scenario is handled in the front end by displaying an error toast to the user informing them of the issue, and assuming there is data available to avoid blocking them. When this occurs, a warning is also logged to the Kibana server logs.  Fixes elastic#200280. ### Notes - Modifying the existing version of the `has_es_data` endpoint in this way should be backward compatible since the behaviour should remain unchanged from before when the client and server versions don't match (please validate if this seems accurate during review). - For a long term fix, the ES team is investigating the issue with `resolve/cluster` and will aim to have it behave like `resolve/index`, which fails quickly when remote clusters are unresponsive. They may also implement other mitigations like a configurable timeout in ES: elastic/elasticsearch#114020. The purpose of this PR is to provide an immediate solution in Kibana that mitigates the issue as much as possible. - If ES ends up providing another performant method for checking if indices exist instead of `resolve/cluster`, Kibana should migrate to that. More details in elastic/elasticsearch#112307. ### Testing notes To reproduce the issue locally, follow these steps: - Follow [these instructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756) to set up a local CCS environment. - Stop the remote cluster process. - Use Netcat on the remote cluster port to listen to requests but not respond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive cluster. See elastic/elasticsearch#32678 for more context. - Navigate to Discover and observe that the `has_es_data` request hangs. When testing in this PR branch, the request will only wait for 5 seconds before assuming data exists and displaying a toast. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_node:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit 96fd4b6)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Nov 20, 2024
…a to hang (elastic#200476) ## Summary This PR mitigates an issue where the `has_es_data` check can hang when some remote clusters are unresponsive, leaving users stuck in a loading state in some apps (e.g. Discover and Dashboard) until the request times out. There are two main changes that help mitigate this issue: - The `resolve/cluster` request in the `has_es_data` endpoint has been split into two requests -- one for local data first, then another for remote data second. In cases where remote clusters are unresponsive but there is data available in the local cluster, the remote check is never performed and the check completes quickly. This likely resolves the majority of cases and is also likely faster in general than checking both local and remote clusters in a single request. - In cases where there is no local data and the remote `resolve/cluster` request hangs, a new `data_views.hasEsDataTimeout` config has been added to `kibana.yml` (defaults to 5 seconds) to abort the request after a short delay. This scenario is handled in the front end by displaying an error toast to the user informing them of the issue, and assuming there is data available to avoid blocking them. When this occurs, a warning is also logged to the Kibana server logs.  Fixes elastic#200280. ### Notes - Modifying the existing version of the `has_es_data` endpoint in this way should be backward compatible since the behaviour should remain unchanged from before when the client and server versions don't match (please validate if this seems accurate during review). - For a long term fix, the ES team is investigating the issue with `resolve/cluster` and will aim to have it behave like `resolve/index`, which fails quickly when remote clusters are unresponsive. They may also implement other mitigations like a configurable timeout in ES: elastic/elasticsearch#114020. The purpose of this PR is to provide an immediate solution in Kibana that mitigates the issue as much as possible. - If ES ends up providing another performant method for checking if indices exist instead of `resolve/cluster`, Kibana should migrate to that. More details in elastic/elasticsearch#112307. ### Testing notes To reproduce the issue locally, follow these steps: - Follow [these instructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756) to set up a local CCS environment. - Stop the remote cluster process. - Use Netcat on the remote cluster port to listen to requests but not respond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive cluster. See elastic/elasticsearch#32678 for more context. - Navigate to Discover and observe that the `has_es_data` request hangs. When testing in this PR branch, the request will only wait for 5 seconds before assuming data exists and displaying a toast. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_node:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit 96fd4b6)
kibanamachine
added a commit
to elastic/kibana
that referenced
this issue
Nov 20, 2024
… can cause Kibana to hang (#200476) (#201025) # Backport This will backport the following commits from `main` to `8.x`: - [[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)](#200476) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Davis McPhee","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-20T18:52:47Z","message":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)\n\n## Summary\r\n\r\nThis PR mitigates an issue where the `has_es_data` check can hang when\r\nsome remote clusters are unresponsive, leaving users stuck in a loading\r\nstate in some apps (e.g. Discover and Dashboard) until the request times\r\nout. There are two main changes that help mitigate this issue:\r\n- The `resolve/cluster` request in the `has_es_data` endpoint has been\r\nsplit into two requests -- one for local data first, then another for\r\nremote data second. In cases where remote clusters are unresponsive but\r\nthere is data available in the local cluster, the remote check is never\r\nperformed and the check completes quickly. This likely resolves the\r\nmajority of cases and is also likely faster in general than checking\r\nboth local and remote clusters in a single request.\r\n- In cases where there is no local data and the remote `resolve/cluster`\r\nrequest hangs, a new `data_views.hasEsDataTimeout` config has been added\r\nto `kibana.yml` (defaults to 5 seconds) to abort the request after a\r\nshort delay. This scenario is handled in the front end by displaying an\r\nerror toast to the user informing them of the issue, and assuming there\r\nis data available to avoid blocking them. When this occurs, a warning is\r\nalso logged to the Kibana server logs.\r\n\r\n\r\n\r\nFixes #200280.\r\n\r\n### Notes\r\n- Modifying the existing version of the `has_es_data` endpoint in this\r\nway should be backward compatible since the behaviour should remain\r\nunchanged from before when the client and server versions don't match\r\n(please validate if this seems accurate during review).\r\n- For a long term fix, the ES team is investigating the issue with\r\n`resolve/cluster` and will aim to have it behave like `resolve/index`,\r\nwhich fails quickly when remote clusters are unresponsive. They may also\r\nimplement other mitigations like a configurable timeout in ES:\r\nhttps://github.com/elastic/elasticsearch/issues/114020. The purpose of\r\nthis PR is to provide an immediate solution in Kibana that mitigates the\r\nissue as much as possible.\r\n- If ES ends up providing another performant method for checking if\r\nindices exist instead of `resolve/cluster`, Kibana should migrate to\r\nthat. More details in\r\nhttps://github.com/elastic/elasticsearch/issues/112307.\r\n\r\n### Testing notes\r\n\r\nTo reproduce the issue locally, follow these steps:\r\n- Follow [these\r\ninstructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756)\r\nto set up a local CCS environment.\r\n- Stop the remote cluster process.\r\n- Use Netcat on the remote cluster port to listen to requests but not\r\nrespond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive\r\ncluster. See elastic/elasticsearch#32678 for\r\nmore context.\r\n- Navigate to Discover and observe that the `has_es_data` request hangs.\r\nWhen testing in this PR branch, the request will only wait for 5 seconds\r\nbefore assuming data exists and displaying a toast.\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [x] This was checked for breaking HTTP API changes, and any breaking\r\nchanges have been approved by the breaking-change committee. The\r\n`release_note:breaking` label should be applied in these situations.\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_node:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"96fd4b682b77f6c1d6d1c6ab0742462d9e9d2589","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v9.0.0","Team:DataDiscovery","backport:prev-major"],"title":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang","number":200476,"url":"https://github.com/elastic/kibana/pull/200476","mergeCommit":{"message":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)\n\n## Summary\r\n\r\nThis PR mitigates an issue where the `has_es_data` check can hang when\r\nsome remote clusters are unresponsive, leaving users stuck in a loading\r\nstate in some apps (e.g. Discover and Dashboard) until the request times\r\nout. There are two main changes that help mitigate this issue:\r\n- The `resolve/cluster` request in the `has_es_data` endpoint has been\r\nsplit into two requests -- one for local data first, then another for\r\nremote data second. In cases where remote clusters are unresponsive but\r\nthere is data available in the local cluster, the remote check is never\r\nperformed and the check completes quickly. This likely resolves the\r\nmajority of cases and is also likely faster in general than checking\r\nboth local and remote clusters in a single request.\r\n- In cases where there is no local data and the remote `resolve/cluster`\r\nrequest hangs, a new `data_views.hasEsDataTimeout` config has been added\r\nto `kibana.yml` (defaults to 5 seconds) to abort the request after a\r\nshort delay. This scenario is handled in the front end by displaying an\r\nerror toast to the user informing them of the issue, and assuming there\r\nis data available to avoid blocking them. When this occurs, a warning is\r\nalso logged to the Kibana server logs.\r\n\r\n\r\n\r\nFixes #200280.\r\n\r\n### Notes\r\n- Modifying the existing version of the `has_es_data` endpoint in this\r\nway should be backward compatible since the behaviour should remain\r\nunchanged from before when the client and server versions don't match\r\n(please validate if this seems accurate during review).\r\n- For a long term fix, the ES team is investigating the issue with\r\n`resolve/cluster` and will aim to have it behave like `resolve/index`,\r\nwhich fails quickly when remote clusters are unresponsive. They may also\r\nimplement other mitigations like a configurable timeout in ES:\r\nhttps://github.com/elastic/elasticsearch/issues/114020. The purpose of\r\nthis PR is to provide an immediate solution in Kibana that mitigates the\r\nissue as much as possible.\r\n- If ES ends up providing another performant method for checking if\r\nindices exist instead of `resolve/cluster`, Kibana should migrate to\r\nthat. More details in\r\nhttps://github.com/elastic/elasticsearch/issues/112307.\r\n\r\n### Testing notes\r\n\r\nTo reproduce the issue locally, follow these steps:\r\n- Follow [these\r\ninstructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756)\r\nto set up a local CCS environment.\r\n- Stop the remote cluster process.\r\n- Use Netcat on the remote cluster port to listen to requests but not\r\nrespond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive\r\ncluster. See elastic/elasticsearch#32678 for\r\nmore context.\r\n- Navigate to Discover and observe that the `has_es_data` request hangs.\r\nWhen testing in this PR branch, the request will only wait for 5 seconds\r\nbefore assuming data exists and displaying a toast.\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [x] This was checked for breaking HTTP API changes, and any breaking\r\nchanges have been approved by the breaking-change committee. The\r\n`release_note:breaking` label should be applied in these situations.\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_node:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"96fd4b682b77f6c1d6d1c6ab0742462d9e9d2589"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/200476","number":200476,"mergeCommit":{"message":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)\n\n## Summary\r\n\r\nThis PR mitigates an issue where the `has_es_data` check can hang when\r\nsome remote clusters are unresponsive, leaving users stuck in a loading\r\nstate in some apps (e.g. Discover and Dashboard) until the request times\r\nout. There are two main changes that help mitigate this issue:\r\n- The `resolve/cluster` request in the `has_es_data` endpoint has been\r\nsplit into two requests -- one for local data first, then another for\r\nremote data second. In cases where remote clusters are unresponsive but\r\nthere is data available in the local cluster, the remote check is never\r\nperformed and the check completes quickly. This likely resolves the\r\nmajority of cases and is also likely faster in general than checking\r\nboth local and remote clusters in a single request.\r\n- In cases where there is no local data and the remote `resolve/cluster`\r\nrequest hangs, a new `data_views.hasEsDataTimeout` config has been added\r\nto `kibana.yml` (defaults to 5 seconds) to abort the request after a\r\nshort delay. This scenario is handled in the front end by displaying an\r\nerror toast to the user informing them of the issue, and assuming there\r\nis data available to avoid blocking them. When this occurs, a warning is\r\nalso logged to the Kibana server logs.\r\n\r\n\r\n\r\nFixes #200280.\r\n\r\n### Notes\r\n- Modifying the existing version of the `has_es_data` endpoint in this\r\nway should be backward compatible since the behaviour should remain\r\nunchanged from before when the client and server versions don't match\r\n(please validate if this seems accurate during review).\r\n- For a long term fix, the ES team is investigating the issue with\r\n`resolve/cluster` and will aim to have it behave like `resolve/index`,\r\nwhich fails quickly when remote clusters are unresponsive. They may also\r\nimplement other mitigations like a configurable timeout in ES:\r\nhttps://github.com/elastic/elasticsearch/issues/114020. The purpose of\r\nthis PR is to provide an immediate solution in Kibana that mitigates the\r\nissue as much as possible.\r\n- If ES ends up providing another performant method for checking if\r\nindices exist instead of `resolve/cluster`, Kibana should migrate to\r\nthat. More details in\r\nhttps://github.com/elastic/elasticsearch/issues/112307.\r\n\r\n### Testing notes\r\n\r\nTo reproduce the issue locally, follow these steps:\r\n- Follow [these\r\ninstructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756)\r\nto set up a local CCS environment.\r\n- Stop the remote cluster process.\r\n- Use Netcat on the remote cluster port to listen to requests but not\r\nrespond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive\r\ncluster. See elastic/elasticsearch#32678 for\r\nmore context.\r\n- Navigate to Discover and observe that the `has_es_data` request hangs.\r\nWhen testing in this PR branch, the request will only wait for 5 seconds\r\nbefore assuming data exists and displaying a toast.\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [x] This was checked for breaking HTTP API changes, and any breaking\r\nchanges have been approved by the breaking-change committee. The\r\n`release_note:breaking` label should be applied in these situations.\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_node:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"96fd4b682b77f6c1d6d1c6ab0742462d9e9d2589"}}]}] BACKPORT--> Co-authored-by: Davis McPhee <[email protected]>
kibanamachine
added a commit
to elastic/kibana
that referenced
this issue
Nov 20, 2024
…k can cause Kibana to hang (#200476) (#201024) # Backport This will backport the following commits from `main` to `8.16`: - [[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)](#200476) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Davis McPhee","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-20T18:52:47Z","message":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)\n\n## Summary\r\n\r\nThis PR mitigates an issue where the `has_es_data` check can hang when\r\nsome remote clusters are unresponsive, leaving users stuck in a loading\r\nstate in some apps (e.g. Discover and Dashboard) until the request times\r\nout. There are two main changes that help mitigate this issue:\r\n- The `resolve/cluster` request in the `has_es_data` endpoint has been\r\nsplit into two requests -- one for local data first, then another for\r\nremote data second. In cases where remote clusters are unresponsive but\r\nthere is data available in the local cluster, the remote check is never\r\nperformed and the check completes quickly. This likely resolves the\r\nmajority of cases and is also likely faster in general than checking\r\nboth local and remote clusters in a single request.\r\n- In cases where there is no local data and the remote `resolve/cluster`\r\nrequest hangs, a new `data_views.hasEsDataTimeout` config has been added\r\nto `kibana.yml` (defaults to 5 seconds) to abort the request after a\r\nshort delay. This scenario is handled in the front end by displaying an\r\nerror toast to the user informing them of the issue, and assuming there\r\nis data available to avoid blocking them. When this occurs, a warning is\r\nalso logged to the Kibana server logs.\r\n\r\n\r\n\r\nFixes #200280.\r\n\r\n### Notes\r\n- Modifying the existing version of the `has_es_data` endpoint in this\r\nway should be backward compatible since the behaviour should remain\r\nunchanged from before when the client and server versions don't match\r\n(please validate if this seems accurate during review).\r\n- For a long term fix, the ES team is investigating the issue with\r\n`resolve/cluster` and will aim to have it behave like `resolve/index`,\r\nwhich fails quickly when remote clusters are unresponsive. They may also\r\nimplement other mitigations like a configurable timeout in ES:\r\nhttps://github.com/elastic/elasticsearch/issues/114020. The purpose of\r\nthis PR is to provide an immediate solution in Kibana that mitigates the\r\nissue as much as possible.\r\n- If ES ends up providing another performant method for checking if\r\nindices exist instead of `resolve/cluster`, Kibana should migrate to\r\nthat. More details in\r\nhttps://github.com/elastic/elasticsearch/issues/112307.\r\n\r\n### Testing notes\r\n\r\nTo reproduce the issue locally, follow these steps:\r\n- Follow [these\r\ninstructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756)\r\nto set up a local CCS environment.\r\n- Stop the remote cluster process.\r\n- Use Netcat on the remote cluster port to listen to requests but not\r\nrespond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive\r\ncluster. See elastic/elasticsearch#32678 for\r\nmore context.\r\n- Navigate to Discover and observe that the `has_es_data` request hangs.\r\nWhen testing in this PR branch, the request will only wait for 5 seconds\r\nbefore assuming data exists and displaying a toast.\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [x] This was checked for breaking HTTP API changes, and any breaking\r\nchanges have been approved by the breaking-change committee. The\r\n`release_note:breaking` label should be applied in these situations.\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_node:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"96fd4b682b77f6c1d6d1c6ab0742462d9e9d2589","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v9.0.0","Team:DataDiscovery","backport:prev-major"],"title":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang","number":200476,"url":"https://github.com/elastic/kibana/pull/200476","mergeCommit":{"message":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)\n\n## Summary\r\n\r\nThis PR mitigates an issue where the `has_es_data` check can hang when\r\nsome remote clusters are unresponsive, leaving users stuck in a loading\r\nstate in some apps (e.g. Discover and Dashboard) until the request times\r\nout. There are two main changes that help mitigate this issue:\r\n- The `resolve/cluster` request in the `has_es_data` endpoint has been\r\nsplit into two requests -- one for local data first, then another for\r\nremote data second. In cases where remote clusters are unresponsive but\r\nthere is data available in the local cluster, the remote check is never\r\nperformed and the check completes quickly. This likely resolves the\r\nmajority of cases and is also likely faster in general than checking\r\nboth local and remote clusters in a single request.\r\n- In cases where there is no local data and the remote `resolve/cluster`\r\nrequest hangs, a new `data_views.hasEsDataTimeout` config has been added\r\nto `kibana.yml` (defaults to 5 seconds) to abort the request after a\r\nshort delay. This scenario is handled in the front end by displaying an\r\nerror toast to the user informing them of the issue, and assuming there\r\nis data available to avoid blocking them. When this occurs, a warning is\r\nalso logged to the Kibana server logs.\r\n\r\n\r\n\r\nFixes #200280.\r\n\r\n### Notes\r\n- Modifying the existing version of the `has_es_data` endpoint in this\r\nway should be backward compatible since the behaviour should remain\r\nunchanged from before when the client and server versions don't match\r\n(please validate if this seems accurate during review).\r\n- For a long term fix, the ES team is investigating the issue with\r\n`resolve/cluster` and will aim to have it behave like `resolve/index`,\r\nwhich fails quickly when remote clusters are unresponsive. They may also\r\nimplement other mitigations like a configurable timeout in ES:\r\nhttps://github.com/elastic/elasticsearch/issues/114020. The purpose of\r\nthis PR is to provide an immediate solution in Kibana that mitigates the\r\nissue as much as possible.\r\n- If ES ends up providing another performant method for checking if\r\nindices exist instead of `resolve/cluster`, Kibana should migrate to\r\nthat. More details in\r\nhttps://github.com/elastic/elasticsearch/issues/112307.\r\n\r\n### Testing notes\r\n\r\nTo reproduce the issue locally, follow these steps:\r\n- Follow [these\r\ninstructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756)\r\nto set up a local CCS environment.\r\n- Stop the remote cluster process.\r\n- Use Netcat on the remote cluster port to listen to requests but not\r\nrespond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive\r\ncluster. See elastic/elasticsearch#32678 for\r\nmore context.\r\n- Navigate to Discover and observe that the `has_es_data` request hangs.\r\nWhen testing in this PR branch, the request will only wait for 5 seconds\r\nbefore assuming data exists and displaying a toast.\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [x] This was checked for breaking HTTP API changes, and any breaking\r\nchanges have been approved by the breaking-change committee. The\r\n`release_note:breaking` label should be applied in these situations.\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_node:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"96fd4b682b77f6c1d6d1c6ab0742462d9e9d2589"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/200476","number":200476,"mergeCommit":{"message":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)\n\n## Summary\r\n\r\nThis PR mitigates an issue where the `has_es_data` check can hang when\r\nsome remote clusters are unresponsive, leaving users stuck in a loading\r\nstate in some apps (e.g. Discover and Dashboard) until the request times\r\nout. There are two main changes that help mitigate this issue:\r\n- The `resolve/cluster` request in the `has_es_data` endpoint has been\r\nsplit into two requests -- one for local data first, then another for\r\nremote data second. In cases where remote clusters are unresponsive but\r\nthere is data available in the local cluster, the remote check is never\r\nperformed and the check completes quickly. This likely resolves the\r\nmajority of cases and is also likely faster in general than checking\r\nboth local and remote clusters in a single request.\r\n- In cases where there is no local data and the remote `resolve/cluster`\r\nrequest hangs, a new `data_views.hasEsDataTimeout` config has been added\r\nto `kibana.yml` (defaults to 5 seconds) to abort the request after a\r\nshort delay. This scenario is handled in the front end by displaying an\r\nerror toast to the user informing them of the issue, and assuming there\r\nis data available to avoid blocking them. When this occurs, a warning is\r\nalso logged to the Kibana server logs.\r\n\r\n\r\n\r\nFixes #200280.\r\n\r\n### Notes\r\n- Modifying the existing version of the `has_es_data` endpoint in this\r\nway should be backward compatible since the behaviour should remain\r\nunchanged from before when the client and server versions don't match\r\n(please validate if this seems accurate during review).\r\n- For a long term fix, the ES team is investigating the issue with\r\n`resolve/cluster` and will aim to have it behave like `resolve/index`,\r\nwhich fails quickly when remote clusters are unresponsive. They may also\r\nimplement other mitigations like a configurable timeout in ES:\r\nhttps://github.com/elastic/elasticsearch/issues/114020. The purpose of\r\nthis PR is to provide an immediate solution in Kibana that mitigates the\r\nissue as much as possible.\r\n- If ES ends up providing another performant method for checking if\r\nindices exist instead of `resolve/cluster`, Kibana should migrate to\r\nthat. More details in\r\nhttps://github.com/elastic/elasticsearch/issues/112307.\r\n\r\n### Testing notes\r\n\r\nTo reproduce the issue locally, follow these steps:\r\n- Follow [these\r\ninstructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756)\r\nto set up a local CCS environment.\r\n- Stop the remote cluster process.\r\n- Use Netcat on the remote cluster port to listen to requests but not\r\nrespond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive\r\ncluster. See elastic/elasticsearch#32678 for\r\nmore context.\r\n- Navigate to Discover and observe that the `has_es_data` request hangs.\r\nWhen testing in this PR branch, the request will only wait for 5 seconds\r\nbefore assuming data exists and displaying a toast.\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [x] This was checked for breaking HTTP API changes, and any breaking\r\nchanges have been approved by the breaking-change committee. The\r\n`release_note:breaking` label should be applied in these situations.\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_node:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"96fd4b682b77f6c1d6d1c6ab0742462d9e9d2589"}}]}] BACKPORT--> Co-authored-by: Davis McPhee <[email protected]>
kibanamachine
added a commit
to elastic/kibana
that referenced
this issue
Nov 20, 2024
…k can cause Kibana to hang (#200476) (#201023) # Backport This will backport the following commits from `main` to `8.15`: - [[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)](#200476) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Davis McPhee","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-20T18:52:47Z","message":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)\n\n## Summary\r\n\r\nThis PR mitigates an issue where the `has_es_data` check can hang when\r\nsome remote clusters are unresponsive, leaving users stuck in a loading\r\nstate in some apps (e.g. Discover and Dashboard) until the request times\r\nout. There are two main changes that help mitigate this issue:\r\n- The `resolve/cluster` request in the `has_es_data` endpoint has been\r\nsplit into two requests -- one for local data first, then another for\r\nremote data second. In cases where remote clusters are unresponsive but\r\nthere is data available in the local cluster, the remote check is never\r\nperformed and the check completes quickly. This likely resolves the\r\nmajority of cases and is also likely faster in general than checking\r\nboth local and remote clusters in a single request.\r\n- In cases where there is no local data and the remote `resolve/cluster`\r\nrequest hangs, a new `data_views.hasEsDataTimeout` config has been added\r\nto `kibana.yml` (defaults to 5 seconds) to abort the request after a\r\nshort delay. This scenario is handled in the front end by displaying an\r\nerror toast to the user informing them of the issue, and assuming there\r\nis data available to avoid blocking them. When this occurs, a warning is\r\nalso logged to the Kibana server logs.\r\n\r\n\r\n\r\nFixes #200280.\r\n\r\n### Notes\r\n- Modifying the existing version of the `has_es_data` endpoint in this\r\nway should be backward compatible since the behaviour should remain\r\nunchanged from before when the client and server versions don't match\r\n(please validate if this seems accurate during review).\r\n- For a long term fix, the ES team is investigating the issue with\r\n`resolve/cluster` and will aim to have it behave like `resolve/index`,\r\nwhich fails quickly when remote clusters are unresponsive. They may also\r\nimplement other mitigations like a configurable timeout in ES:\r\nhttps://github.com/elastic/elasticsearch/issues/114020. The purpose of\r\nthis PR is to provide an immediate solution in Kibana that mitigates the\r\nissue as much as possible.\r\n- If ES ends up providing another performant method for checking if\r\nindices exist instead of `resolve/cluster`, Kibana should migrate to\r\nthat. More details in\r\nhttps://github.com/elastic/elasticsearch/issues/112307.\r\n\r\n### Testing notes\r\n\r\nTo reproduce the issue locally, follow these steps:\r\n- Follow [these\r\ninstructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756)\r\nto set up a local CCS environment.\r\n- Stop the remote cluster process.\r\n- Use Netcat on the remote cluster port to listen to requests but not\r\nrespond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive\r\ncluster. See elastic/elasticsearch#32678 for\r\nmore context.\r\n- Navigate to Discover and observe that the `has_es_data` request hangs.\r\nWhen testing in this PR branch, the request will only wait for 5 seconds\r\nbefore assuming data exists and displaying a toast.\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [x] This was checked for breaking HTTP API changes, and any breaking\r\nchanges have been approved by the breaking-change committee. The\r\n`release_note:breaking` label should be applied in these situations.\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_node:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"96fd4b682b77f6c1d6d1c6ab0742462d9e9d2589","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v9.0.0","Team:DataDiscovery","backport:prev-major"],"title":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang","number":200476,"url":"https://github.com/elastic/kibana/pull/200476","mergeCommit":{"message":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)\n\n## Summary\r\n\r\nThis PR mitigates an issue where the `has_es_data` check can hang when\r\nsome remote clusters are unresponsive, leaving users stuck in a loading\r\nstate in some apps (e.g. Discover and Dashboard) until the request times\r\nout. There are two main changes that help mitigate this issue:\r\n- The `resolve/cluster` request in the `has_es_data` endpoint has been\r\nsplit into two requests -- one for local data first, then another for\r\nremote data second. In cases where remote clusters are unresponsive but\r\nthere is data available in the local cluster, the remote check is never\r\nperformed and the check completes quickly. This likely resolves the\r\nmajority of cases and is also likely faster in general than checking\r\nboth local and remote clusters in a single request.\r\n- In cases where there is no local data and the remote `resolve/cluster`\r\nrequest hangs, a new `data_views.hasEsDataTimeout` config has been added\r\nto `kibana.yml` (defaults to 5 seconds) to abort the request after a\r\nshort delay. This scenario is handled in the front end by displaying an\r\nerror toast to the user informing them of the issue, and assuming there\r\nis data available to avoid blocking them. When this occurs, a warning is\r\nalso logged to the Kibana server logs.\r\n\r\n\r\n\r\nFixes #200280.\r\n\r\n### Notes\r\n- Modifying the existing version of the `has_es_data` endpoint in this\r\nway should be backward compatible since the behaviour should remain\r\nunchanged from before when the client and server versions don't match\r\n(please validate if this seems accurate during review).\r\n- For a long term fix, the ES team is investigating the issue with\r\n`resolve/cluster` and will aim to have it behave like `resolve/index`,\r\nwhich fails quickly when remote clusters are unresponsive. They may also\r\nimplement other mitigations like a configurable timeout in ES:\r\nhttps://github.com/elastic/elasticsearch/issues/114020. The purpose of\r\nthis PR is to provide an immediate solution in Kibana that mitigates the\r\nissue as much as possible.\r\n- If ES ends up providing another performant method for checking if\r\nindices exist instead of `resolve/cluster`, Kibana should migrate to\r\nthat. More details in\r\nhttps://github.com/elastic/elasticsearch/issues/112307.\r\n\r\n### Testing notes\r\n\r\nTo reproduce the issue locally, follow these steps:\r\n- Follow [these\r\ninstructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756)\r\nto set up a local CCS environment.\r\n- Stop the remote cluster process.\r\n- Use Netcat on the remote cluster port to listen to requests but not\r\nrespond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive\r\ncluster. See elastic/elasticsearch#32678 for\r\nmore context.\r\n- Navigate to Discover and observe that the `has_es_data` request hangs.\r\nWhen testing in this PR branch, the request will only wait for 5 seconds\r\nbefore assuming data exists and displaying a toast.\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [x] This was checked for breaking HTTP API changes, and any breaking\r\nchanges have been approved by the breaking-change committee. The\r\n`release_note:breaking` label should be applied in these situations.\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_node:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"96fd4b682b77f6c1d6d1c6ab0742462d9e9d2589"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/200476","number":200476,"mergeCommit":{"message":"[Data Views] Mitigate issue where `has_es_data` check can cause Kibana to hang (#200476)\n\n## Summary\r\n\r\nThis PR mitigates an issue where the `has_es_data` check can hang when\r\nsome remote clusters are unresponsive, leaving users stuck in a loading\r\nstate in some apps (e.g. Discover and Dashboard) until the request times\r\nout. There are two main changes that help mitigate this issue:\r\n- The `resolve/cluster` request in the `has_es_data` endpoint has been\r\nsplit into two requests -- one for local data first, then another for\r\nremote data second. In cases where remote clusters are unresponsive but\r\nthere is data available in the local cluster, the remote check is never\r\nperformed and the check completes quickly. This likely resolves the\r\nmajority of cases and is also likely faster in general than checking\r\nboth local and remote clusters in a single request.\r\n- In cases where there is no local data and the remote `resolve/cluster`\r\nrequest hangs, a new `data_views.hasEsDataTimeout` config has been added\r\nto `kibana.yml` (defaults to 5 seconds) to abort the request after a\r\nshort delay. This scenario is handled in the front end by displaying an\r\nerror toast to the user informing them of the issue, and assuming there\r\nis data available to avoid blocking them. When this occurs, a warning is\r\nalso logged to the Kibana server logs.\r\n\r\n\r\n\r\nFixes #200280.\r\n\r\n### Notes\r\n- Modifying the existing version of the `has_es_data` endpoint in this\r\nway should be backward compatible since the behaviour should remain\r\nunchanged from before when the client and server versions don't match\r\n(please validate if this seems accurate during review).\r\n- For a long term fix, the ES team is investigating the issue with\r\n`resolve/cluster` and will aim to have it behave like `resolve/index`,\r\nwhich fails quickly when remote clusters are unresponsive. They may also\r\nimplement other mitigations like a configurable timeout in ES:\r\nhttps://github.com/elastic/elasticsearch/issues/114020. The purpose of\r\nthis PR is to provide an immediate solution in Kibana that mitigates the\r\nissue as much as possible.\r\n- If ES ends up providing another performant method for checking if\r\nindices exist instead of `resolve/cluster`, Kibana should migrate to\r\nthat. More details in\r\nhttps://github.com/elastic/elasticsearch/issues/112307.\r\n\r\n### Testing notes\r\n\r\nTo reproduce the issue locally, follow these steps:\r\n- Follow [these\r\ninstructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756)\r\nto set up a local CCS environment.\r\n- Stop the remote cluster process.\r\n- Use Netcat on the remote cluster port to listen to requests but not\r\nrespond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive\r\ncluster. See elastic/elasticsearch#32678 for\r\nmore context.\r\n- Navigate to Discover and observe that the `has_es_data` request hangs.\r\nWhen testing in this PR branch, the request will only wait for 5 seconds\r\nbefore assuming data exists and displaying a toast.\r\n\r\n### Checklist\r\n\r\n- [x] Any text added follows [EUI's writing\r\nguidelines](https://elastic.github.io/eui/#/guidelines/writing), uses\r\nsentence case text and includes [i18n\r\nsupport](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)\r\n- [ ]\r\n[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)\r\nwas added for features that require explanation or tutorials\r\n- [x] [Unit or functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere updated or added to match the most common scenarios\r\n- [ ] If a plugin configuration key changed, check if it needs to be\r\nallowlisted in the cloud and added to the [docker\r\nlist](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)\r\n- [x] This was checked for breaking HTTP API changes, and any breaking\r\nchanges have been approved by the breaking-change committee. The\r\n`release_note:breaking` label should be applied in these situations.\r\n- [ ] [Flaky Test\r\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\r\nused on any tests changed\r\n- [x] The PR description includes the appropriate Release Notes section,\r\nand the correct `release_node:*` label is applied per the\r\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <[email protected]>","sha":"96fd4b682b77f6c1d6d1c6ab0742462d9e9d2589"}}]}] BACKPORT--> --------- Co-authored-by: Davis McPhee <[email protected]>
TattdCodeMonkey
pushed a commit
to TattdCodeMonkey/kibana
that referenced
this issue
Nov 21, 2024
…a to hang (elastic#200476) ## Summary This PR mitigates an issue where the `has_es_data` check can hang when some remote clusters are unresponsive, leaving users stuck in a loading state in some apps (e.g. Discover and Dashboard) until the request times out. There are two main changes that help mitigate this issue: - The `resolve/cluster` request in the `has_es_data` endpoint has been split into two requests -- one for local data first, then another for remote data second. In cases where remote clusters are unresponsive but there is data available in the local cluster, the remote check is never performed and the check completes quickly. This likely resolves the majority of cases and is also likely faster in general than checking both local and remote clusters in a single request. - In cases where there is no local data and the remote `resolve/cluster` request hangs, a new `data_views.hasEsDataTimeout` config has been added to `kibana.yml` (defaults to 5 seconds) to abort the request after a short delay. This scenario is handled in the front end by displaying an error toast to the user informing them of the issue, and assuming there is data available to avoid blocking them. When this occurs, a warning is also logged to the Kibana server logs.  Fixes elastic#200280. ### Notes - Modifying the existing version of the `has_es_data` endpoint in this way should be backward compatible since the behaviour should remain unchanged from before when the client and server versions don't match (please validate if this seems accurate during review). - For a long term fix, the ES team is investigating the issue with `resolve/cluster` and will aim to have it behave like `resolve/index`, which fails quickly when remote clusters are unresponsive. They may also implement other mitigations like a configurable timeout in ES: elastic/elasticsearch#114020. The purpose of this PR is to provide an immediate solution in Kibana that mitigates the issue as much as possible. - If ES ends up providing another performant method for checking if indices exist instead of `resolve/cluster`, Kibana should migrate to that. More details in elastic/elasticsearch#112307. ### Testing notes To reproduce the issue locally, follow these steps: - Follow [these instructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756) to set up a local CCS environment. - Stop the remote cluster process. - Use Netcat on the remote cluster port to listen to requests but not respond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive cluster. See elastic/elasticsearch#32678 for more context. - Navigate to Discover and observe that the `has_es_data` request hangs. When testing in this PR branch, the request will only wait for 5 seconds before assuming data exists and displaying a toast. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_node:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]>
paulinashakirova
pushed a commit
to paulinashakirova/kibana
that referenced
this issue
Nov 26, 2024
…a to hang (elastic#200476) ## Summary This PR mitigates an issue where the `has_es_data` check can hang when some remote clusters are unresponsive, leaving users stuck in a loading state in some apps (e.g. Discover and Dashboard) until the request times out. There are two main changes that help mitigate this issue: - The `resolve/cluster` request in the `has_es_data` endpoint has been split into two requests -- one for local data first, then another for remote data second. In cases where remote clusters are unresponsive but there is data available in the local cluster, the remote check is never performed and the check completes quickly. This likely resolves the majority of cases and is also likely faster in general than checking both local and remote clusters in a single request. - In cases where there is no local data and the remote `resolve/cluster` request hangs, a new `data_views.hasEsDataTimeout` config has been added to `kibana.yml` (defaults to 5 seconds) to abort the request after a short delay. This scenario is handled in the front end by displaying an error toast to the user informing them of the issue, and assuming there is data available to avoid blocking them. When this occurs, a warning is also logged to the Kibana server logs.  Fixes elastic#200280. ### Notes - Modifying the existing version of the `has_es_data` endpoint in this way should be backward compatible since the behaviour should remain unchanged from before when the client and server versions don't match (please validate if this seems accurate during review). - For a long term fix, the ES team is investigating the issue with `resolve/cluster` and will aim to have it behave like `resolve/index`, which fails quickly when remote clusters are unresponsive. They may also implement other mitigations like a configurable timeout in ES: elastic/elasticsearch#114020. The purpose of this PR is to provide an immediate solution in Kibana that mitigates the issue as much as possible. - If ES ends up providing another performant method for checking if indices exist instead of `resolve/cluster`, Kibana should migrate to that. More details in elastic/elasticsearch#112307. ### Testing notes To reproduce the issue locally, follow these steps: - Follow [these instructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756) to set up a local CCS environment. - Stop the remote cluster process. - Use Netcat on the remote cluster port to listen to requests but not respond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive cluster. See elastic/elasticsearch#32678 for more context. - Navigate to Discover and observe that the `has_es_data` request hangs. When testing in this PR branch, the request will only wait for 5 seconds before assuming data exists and displaying a toast. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_node:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]>
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Dec 12, 2024
…a to hang (elastic#200476) ## Summary This PR mitigates an issue where the `has_es_data` check can hang when some remote clusters are unresponsive, leaving users stuck in a loading state in some apps (e.g. Discover and Dashboard) until the request times out. There are two main changes that help mitigate this issue: - The `resolve/cluster` request in the `has_es_data` endpoint has been split into two requests -- one for local data first, then another for remote data second. In cases where remote clusters are unresponsive but there is data available in the local cluster, the remote check is never performed and the check completes quickly. This likely resolves the majority of cases and is also likely faster in general than checking both local and remote clusters in a single request. - In cases where there is no local data and the remote `resolve/cluster` request hangs, a new `data_views.hasEsDataTimeout` config has been added to `kibana.yml` (defaults to 5 seconds) to abort the request after a short delay. This scenario is handled in the front end by displaying an error toast to the user informing them of the issue, and assuming there is data available to avoid blocking them. When this occurs, a warning is also logged to the Kibana server logs.  Fixes elastic#200280. ### Notes - Modifying the existing version of the `has_es_data` endpoint in this way should be backward compatible since the behaviour should remain unchanged from before when the client and server versions don't match (please validate if this seems accurate during review). - For a long term fix, the ES team is investigating the issue with `resolve/cluster` and will aim to have it behave like `resolve/index`, which fails quickly when remote clusters are unresponsive. They may also implement other mitigations like a configurable timeout in ES: elastic/elasticsearch#114020. The purpose of this PR is to provide an immediate solution in Kibana that mitigates the issue as much as possible. - If ES ends up providing another performant method for checking if indices exist instead of `resolve/cluster`, Kibana should migrate to that. More details in elastic/elasticsearch#112307. ### Testing notes To reproduce the issue locally, follow these steps: - Follow [these instructions](https://gist.github.com/lukasolson/d0861aa3e6ee476ac8dd7189ed476756) to set up a local CCS environment. - Stop the remote cluster process. - Use Netcat on the remote cluster port to listen to requests but not respond (e.g. on macOS: `nc -l 9600`), simulating an unresponsive cluster. See elastic/elasticsearch#32678 for more context. - Navigate to Discover and observe that the `has_es_data` request hangs. When testing in this PR branch, the request will only wait for 5 seconds before assuming data exists and displaying a toast. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] The PR description includes the appropriate Release Notes section, and the correct `release_node:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) --------- Co-authored-by: kibanamachine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Kibana needs to determine whether it should display a "Getting started" flow in a number of places, which is done with this utility - https://github.com/elastic/kibana/blob/main/src/plugins/data_views/public/services/has_data.ts
The resolve/index api doesn't scale since it always returns ALL the indices. This can be extremely slow for large and complicated deployments - https://github.com/elastic/sdh-kibana/issues/4877
In my tests, using the search API for this is also relatively slow. This functionality should have a sub second response time.
The resolve/cluster api works well for this BUT its not available on serverless. As a compromise, we're using the resolve/cluster API, falling back to resolve/index as needed.
Ideally, the API would take an index pattern with exclusions and simply return a boolean. We really don't need the list of matched indices. In theory, determining if there is user data should be easy but in practice we have a list of indices that are created by kibana that we don't consider to be user data. I don't entirely understand why this is the case but it would take an organizational effort to resolve it.
The text was updated successfully, but these errors were encountered: