[Console] Generate autocomplete definitions from ES specification #163301

yuliacech · 2023-08-07T13:56:27Z

Summary

This PR uses the new script to generate autocomplete definitions for Dev Tools Console from the ES specification repo.

Definitions changes

New property availability is added to filter out endpoints that are not available in Serverless
Some endpoints' query parameters have more details now, for example common query params are now defined in definitions

"url_params": {
      "error_trace": "__flag__",
      "filter_path": [],
      "human": "__flag__",
      "pretty": "__flag__"
},

Url components in few endpoints are removed, but those were added to overrides files in [Console] Generate url components with the new script #163096
Documentation links contain {branch} instead of master (fix for that added in [Console] Use ES specification for autocomplete definitions #159241)

Script changes

The logic for generating availability for endpoint has been updated based on the feedback from the Clients team. Details added to the script file.
Added a few "safe guards" to the spots in the script where an unexpected type of data might be coming from the ES specification schema

Console changes

Fixed the autocomplete request on Serverless (we might need a proper fix for that, details in [Console] Autocomplete suggestions are not displayed on Serverless #163318)

Also updates to readme files both in Console and the new script.
I will remove the old script in a separate PR.

Screenshots

"ILM" autocomplete suggestions displayed on stateful

"ILM" autocomplete suggestions not displayed on serverless

How to test

Start Kibana on stateful (yarn start) and check that autocomplete suggestions are working as before (no changes)
Start Kibana on serverless (yarn start --serverless) and check that autocomplete suggestions are not displayed for endpoints blocked/internal on serverless.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

…m the Clients team

yuliacech · 2023-08-07T15:52:37Z

packages/kbn-generate-console-definitions/src/generate_availability.ts

-    availability.stack = endpoint.availability.stack?.visibility === 'public';
+  // availability is a required property of the endpoint
+  if (!endpoint.availability) {
+    throw new Error('missing availability for ' + endpoint.name);


Throwing an error here because the script expects all endpoints to have availability according to types. An error here will indicate that something changed in the schema.

What do you think about adding a test to expect this scenario to throw? 🤔 I've checked the .test.ts file but couldnt find any for this scenario

Good idea, will add a test for this!

yuliacech · 2023-08-07T15:53:36Z

packages/kbn-generate-console-definitions/src/helpers/convert_url_properties.ts

@@ -47,7 +47,7 @@ const convertValueOf = (
    return convertLiteralValue(valueOf);
  }
  // for query params we can ignore 'dictionary_of' and 'user_defined_value'
-  return '';
+  throw new Error('unexpected valueOf type ' + kind);


another safeguard: the script doesn't expect types like dictionary and literal value for url params. an error here will indicate that an update is needed.

yuliacech · 2023-08-07T15:54:12Z

src/plugins/console/public/lib/kb/kb.js

@@ -124,6 +124,8 @@ export function setActiveApi(api) {
      dataType: 'json', // disable automatic guessing
      headers: {
        'kbn-xsrf': 'kibana',
+        // workaround for serverless
+        'x-elastic-internal-origin': 'Kibana',


a workaround to fix #163318, needs to be refactored to the Kibana http client

I understand this isn't the preferred approach, but just to confirm - are there any particular concerns with moving forward with this workaround for now?

No concerns, I confirmed with the Core team that adding the header is one way to fix this issue. There are more details in this PR. My only concern is that we use jQuery ajax for this request 😓

elasticmachine · 2023-08-07T16:44:35Z

Pinging @elastic/platform-deployment-management (Team:Deployment Management)

alisonelizabeth

Great work @yuliacech 🎉 Left a few questions in the code about some of the generated definitions.

I did some smoke testing locally. Everything seemed to work as expected for stateful. For serverless, I am seeing some autocomplete suggestions that I would not expect - I think these are coming from the overrides directory.

For example:

alisonelizabeth · 2023-08-09T12:42:21Z

src/plugins/console/public/lib/kb/kb.js

@@ -124,6 +124,8 @@ export function setActiveApi(api) {
      dataType: 'json', // disable automatic guessing
      headers: {
        'kbn-xsrf': 'kibana',
+        // workaround for serverless
+        'x-elastic-internal-origin': 'Kibana',


I understand this isn't the preferred approach, but just to confirm - are there any particular concerns with moving forward with this workaround for now?

alisonelizabeth · 2023-08-09T12:53:14Z

src/plugins/console/server/lib/spec_definitions/json/generated/cluster.stats.json

+    "documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/cluster-stats.html",
+    "availability": {
+      "stack": true,
+      "serverless": true


I'm surprised this one is public in serverless

Looks like this needs to be updated in the spec, it's marked as public for serverless there

alisonelizabeth · 2023-08-09T12:59:25Z

src/plugins/console/server/lib/spec_definitions/json/generated/indices.delete_template.json

+    "documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-templates.html",
+    "availability": {
+      "stack": true,
+      "serverless": true


This is for legacy index templates, right? I thought that was going to be blocked for serverless.

This needs update in the spec too, marked as public for serverless

alisonelizabeth · 2023-08-09T13:22:00Z

src/plugins/console/server/lib/spec_definitions/json/generated/ml.validate.json

    "methods": [
      "POST"
    ],
    "patterns": [
      "_ml/anomaly_detectors/_validate"
    ],
-    "documentation": "https://www.elastic.co/guide/en/machine-learning/current/ml-jobs.html"
+    "documentation": "https://www.elastic.co/guide/en/machine-learning/current/ml-jobs.html",
+    "availability": {


What does it mean if the API availability is false in both environments?

Usually that is either an internal endpoint like for example _internal.delete_desired_balance, or it's still under a feature flag, or in this case it looks like it's a deprecated/moved/renamed endpoint. I was not able to find this one in the docs pages (link).

Thanks for the explanation. It sounds like this one may also need to be updated in the specs then?

Yes, I think this one is probably moved to a different endpoint or deprecated completely. This should be updated in the spec.

alisonelizabeth · 2023-08-09T13:26:34Z

src/plugins/console/server/lib/spec_definitions/json/generated/search_application.list.json

    },
    "methods": [
      "GET"
    ],
    "patterns": [
      "_application/search_application"
    ],
-    "documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/master/list-search-applications.html"
+    "documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/master/list-search-applications.html",


Is there a concept of APIs specific to a serverless project type?

I don't think so, I asked about that the ES team but they said for now it's just public vs internal vs blocked in serverless. And the ES spec only has stack vs serverless availability

alisonelizabeth · 2023-08-09T13:38:40Z

src/plugins/console/server/lib/spec_definitions/json/generated/indices.disk_usage.json

+    "documentation": "https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-disk-usage.html",
+    "availability": {
+      "stack": true,
+      "serverless": true


I'm surprised this one is public too on serverless

yes, this one is "internal" in the sheet but not yet updated in the spec

sabarasaba

Nice work @yuliacech! Did small bit of local testing and autocompletion seems to be correct. Left one small comment, let me know!

sabarasaba · 2023-08-14T13:43:27Z

packages/kbn-generate-console-definitions/src/generate_availability.ts

-    availability.stack = endpoint.availability.stack?.visibility === 'public';
+  // availability is a required property of the endpoint
+  if (!endpoint.availability) {
+    throw new Error('missing availability for ' + endpoint.name);


What do you think about adding a test to expect this scenario to throw? 🤔 I've checked the .test.ts file but couldnt find any for this scenario

alisonelizabeth

Latest LGTM. Thanks for addressing my feedback!

yuliacech · 2023-08-14T15:35:01Z

Thanks a lot for the reviews, @alisonelizabeth @sabarasaba!
@alisonelizabeth Great catch with the wrong endpoint names, fixed in 5e13a64. Those errors were introduced in #162652 and I opened #163611 to add more checks and prevent this in the future.
@sabarasaba Great suggestions about the test, added in 9ea0bab

yuliacech · 2023-08-14T15:43:58Z

There is a PR open in the ES specification repo that updates the availability in serverless. After this PR is merged we could re-generate the definitions some time in the future when the spec PR is merged and the changes are propagated to the schema. This should be easy to do with the script and the changes will probably not be too extensive.

kibana-ci · 2023-08-14T16:52:47Z

💚 Build Succeeded

Buildkite Build
Commit: 237f4a5

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`console`	416.6KB	416.7KB	+37.0B

History

💚 Build #150029 succeeded 89eb6fd
💚 Build #149222 succeeded eef91cc
💚 Build #148041 succeeded 22da200

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…63301) ## Summary This PR uses the new script to generate autocomplete definitions for Dev Tools Console from the ES specification repo. #### Definitions changes - New property `availability` is added to filter out endpoints that are not available in Serverless - Some endpoints' query parameters have more details now, for example common query params are now defined in definitions ```json "url_params": { "error_trace": "__flag__", "filter_path": [], "human": "__flag__", "pretty": "__flag__" }, ``` - Url components in few endpoints are removed, but those were added to overrides files in #163096 - Documentation links contain `{branch}` instead of `master` (fix for that added in #159241) #### Script changes - The logic for generating `availability` for endpoint has been updated based on the feedback from the Clients team. Details added to the script file. - Added a few "safe guards" to the spots in the script where an unexpected type of data might be coming from the ES specification schema #### Console changes - Fixed the autocomplete request on Serverless (we might need a proper fix for that, details in #163318) Also updates to readme files both in Console and the new script. I will remove the old script in a separate PR. ## Screenshots "ILM" autocomplete suggestions displayed on stateful <img width="583" alt="Screenshot 2023-08-07 at 17 47 48" src="https://github.com/elastic/kibana/assets/6585477/641a48b0-fb1a-4d3b-a8c9-99eab8795510"> "ILM" autocomplete suggestions not displayed on serverless <img width="572" alt="Screenshot 2023-08-07 at 17 35 16" src="https://github.com/elastic/kibana/assets/6585477/a1ee5468-eb9f-4f52-81d5-c661b06f8ceb"> ## How to test - Start Kibana on stateful (`yarn start`) and check that autocomplete suggestions are working as before (no changes) - Start Kibana on serverless (`yarn start --serverless`) and check that autocomplete suggestions are not displayed for endpoints blocked/internal on serverless. ### Checklist Delete any items that are not applicable to this PR. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [ ] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers)

yuliacech added 4 commits August 7, 2023 13:06

[Console] Update the availability defaults

4cb1362

[Console] Generate autocomplete definitions with the new script

32d0c70

[Console] Update the availability generation logic after feedback fro…

e36b095

…m the Clients team

[Console] Add a fail safe for "unexpected" url params

a40c1b9

yuliacech added Feature:Console Dev Tools Console Feature Team:Kibana Management Dev Tools, Index Management, Upgrade Assistant, ILM, Ingest Node Pipelines, and more release_note:skip Skip the PR/issue when compiling release notes v8.10.0 labels Aug 7, 2023

yuliacech added 3 commits August 7, 2023 16:13

[Console] Update the readme for console

9abbf0e

[Console] Update the script readme

aedd77b

[Console] Fix the autocomplete endpoint failing on serverless

1c2ec83

yuliacech commented Aug 7, 2023

View reviewed changes

Merge branch 'main' into console/generate_availability_new_definitions

22da200

yuliacech mentioned this pull request Aug 7, 2023

[Console] Remove the old script for definitions generation #163333

Closed

yuliacech marked this pull request as ready for review August 7, 2023 16:44

yuliacech requested a review from a team as a code owner August 7, 2023 16:44

yuliacech requested review from sabarasaba and alisonelizabeth August 8, 2023 16:34

alisonelizabeth reviewed Aug 9, 2023

View reviewed changes

yuliacech and others added 3 commits August 10, 2023 15:29

[Console] Fix the wrong endpoint names in the override files

5e13a64

Merge branch 'main' into console/generate_availability_new_definitions

eef91cc

Merge branch 'main' into console/generate_availability_new_definitions

89eb6fd

sabarasaba reviewed Aug 14, 2023

View reviewed changes

alisonelizabeth approved these changes Aug 14, 2023

View reviewed changes

[Console] Add a test for error when an endpoint is missing availability

9ea0bab

[Console] Fix ts error

237f4a5

sabarasaba approved these changes Aug 14, 2023

View reviewed changes

yuliacech merged commit 11e57be into elastic:main Aug 15, 2023

kibanamachine added the backport:skip This commit does not require backporting label Aug 15, 2023

yuliacech mentioned this pull request Aug 17, 2023

[Console] Use Elasticsearch specification repo for autocomplete definitions #130553

Closed

alisonelizabeth mentioned this pull request Aug 29, 2023

[Console] Autocomplete suggestions are not displayed on Serverless #163318

Open

yuliacech deleted the console/generate_availability_new_definitions branch February 15, 2024 12:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Console] Generate autocomplete definitions from ES specification #163301

[Console] Generate autocomplete definitions from ES specification #163301

yuliacech commented Aug 7, 2023 •

edited

Loading

yuliacech Aug 7, 2023

sabarasaba Aug 14, 2023

yuliacech Aug 14, 2023

yuliacech Aug 7, 2023

yuliacech Aug 7, 2023

alisonelizabeth Aug 9, 2023

yuliacech Aug 10, 2023

elasticmachine commented Aug 7, 2023

alisonelizabeth left a comment

alisonelizabeth Aug 9, 2023

alisonelizabeth Aug 9, 2023

yuliacech Aug 10, 2023

alisonelizabeth Aug 9, 2023

yuliacech Aug 10, 2023

alisonelizabeth Aug 9, 2023 •

edited

Loading

yuliacech Aug 10, 2023

alisonelizabeth Aug 14, 2023

yuliacech Aug 14, 2023

alisonelizabeth Aug 9, 2023

yuliacech Aug 10, 2023

alisonelizabeth Aug 9, 2023

yuliacech Aug 10, 2023

sabarasaba left a comment

sabarasaba Aug 14, 2023

alisonelizabeth left a comment

yuliacech commented Aug 14, 2023

yuliacech commented Aug 14, 2023

kibana-ci commented Aug 14, 2023

[Console] Generate autocomplete definitions from ES specification #163301

[Console] Generate autocomplete definitions from ES specification #163301

Conversation

yuliacech commented Aug 7, 2023 • edited Loading

Summary

Definitions changes

Script changes

Console changes

Screenshots

How to test

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Aug 7, 2023

alisonelizabeth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alisonelizabeth Aug 9, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sabarasaba left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alisonelizabeth left a comment

Choose a reason for hiding this comment

yuliacech commented Aug 14, 2023

yuliacech commented Aug 14, 2023

kibana-ci commented Aug 14, 2023

💚 Build Succeeded

Metrics [docs]

Async chunks

History

yuliacech commented Aug 7, 2023 •

edited

Loading

alisonelizabeth Aug 9, 2023 •

edited

Loading