Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ESQL] Add caching for ESQL validation and autocomplete requests #171866

Merged
merged 6 commits into from
Nov 27, 2023
Merged

[ESQL] Add caching for ESQL validation and autocomplete requests #171866

merged 6 commits into from
Nov 27, 2023

Conversation

dej611
Copy link
Contributor

@dej611 dej611 commented Nov 23, 2023

Summary

This PR adds a cache at query level refreshed every 10 minutes for fields queries used in both validation and autocomplete.

Checklist

@dej611 dej611 added Team:Visualizations Visualization editors, elastic-charts and infrastructure release_note:skip Skip the PR/issue when compiling release notes Feature:ES|QL ES|QL related features in Kibana v8.12.0 labels Nov 23, 2023
@dej611 dej611 marked this pull request as ready for review November 24, 2023 11:47
@dej611 dej611 requested review from a team as code owners November 24, 2023 11:47
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-visualizations (Team:Visualizations)

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] Serverless Investigations - Security Solution Cypress Tests #8 / Detections : Page Filters Filters are restored from localstorage when user navigates back to the page. Filters are restored from localstorage when user navigates back to the page.
  • [job] [logs] FTR Configs #31 / serverless observability UI Rules list should render percentile column and cells correctly

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 12.8MB 12.8MB +143.0B
stackAlerts 203.0KB 203.0KB +45.0B
textBasedLanguages 151.0KB 151.4KB +362.0B
total +550.0B
Unknown metric groups

ESLint disabled line counts

id before after diff
@kbn/text-based-editor 3 2 -1

Total ESLint disabled count

id before after diff
@kbn/text-based-editor 3 2 -1

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

stratoula
stratoula reviewed Nov 27, 2023
View reviewed changes
Copy link
Contributor

@stratoula stratoula left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanx Marco. This looks great. Only 2 comments:

  • One comment irrelevant with the cache. When there are warnings I don't want the code to be highlighted, only the warning icon is sufficient.
image
  • Can we update this PR in order to also have the autocomplete to see how it works there too?

@dej611
Copy link
Contributor Author

dej611 commented Nov 27, 2023

Can we update this PR in order to also have the autocomplete to see how it works there too?

It should be up to date already. 🤔

One comment irrelevant with the cache. When there are warnings I don't want the code to be highlighted, only the warning icon is sufficient.

In your screenshot I see a warning for a message without exact location, so by default the highlight is for the first 10 chars of the query. Do you mean specifically to disable the highlight for this type of warnings or for all warnings?

@stratoula
Copy link
Contributor

In your screenshot I see a warning for a message without exact location, so by default the highlight is for the first 10 chars of the query. Do you mean specifically to disable the highlight for this type of warnings or for all warnings?

For all warnings as it is in main now

It should be up to date already. 🤔

Hmm autocomplete didn't work for me but I can try again.

@dej611
Copy link
Contributor Author

dej611 commented Nov 27, 2023 •
edited
Loading

For all warnings as it is in main now

I'll add a note on the feature branch. I can see some value in specific highlight nonetheless (i.e. divide by zero, drop @timestamp, etc...) but we can get them back later.

@dej611
Copy link
Contributor Author

dej611 commented Nov 27, 2023

Added PR for the warning bit here: #171968

@dej611 dej611 mentioned this pull request Nov 27, 2023
Merged
23 tasks
stratoula
stratoula approved these changes Nov 27, 2023
View reviewed changes
Copy link
Contributor

@stratoula stratoula left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works great (I bootstrapped again). LGTM!

@dej611 dej611 merged commit 82d66ed into elastic:feature/esql-validation Nov 27, 2023
1 check passed
dej611 added a commit that referenced this pull request Dec 7, 2023
@dej611 @kibanamachine
@stratoula @abdonpijpelink
[ES|QL] Advanced validation and autocomplete (#170071)
ff49526 
## Summary

Fixes #166242 , #166876, #166173
, part of #166092, #166084

List of tasks:

* [x] AST work ( #166185 )
* [x] Basic validate work ( #166185 )
* [x] Hover feature ( #166185 )
* [x] Initial autocomplete work with new AST ( #166185 )
* Complete validation feature for MVP
  * [x] wildcards support ( #170014 )
  * [x] remote index validation support ( #171996 )
  * [x] wildcard support as `count` argument  ( #172054 )
  * [x] Aggressive caching for field queries:
* cache as much as possible the `FROM` queries - possible clear the
cache every 10/15 minutes?
    * do not fire a query when code == submitted code
* cache as much as possible the custom `FROM` built from `ENRICH`
policies - same clear policy as above
  * [x] Add unsupported fields warning messages
  * [x] Notify usage of `project` command with deprecation `warning`
* Complete autocomplete work ( #171664 )
  * [x] `stats`
  * [x] `where`
  * [x] `eval`
    * `math syntax`
  * [x] Aggressive cache for fields queries? ( #171866 )
  * [x] Fix when cursor is not at the end position ( #172060 ) 
* [x] Revisit copy messages
  * Label Kibana-only messages
* [x] Extend hover feature ( #171940 )
* [x] Disable editor query highlight for warnings ( #171968 )
* Fix editor highlight with new grammar
  * [x] on multi-line ( #172080 )
  * [x] for functions ( #172287 )

## Release notes

Enhance ES|QL query editing experience with client side validation.
Enhance ES|QL suggestions experience with more in context suggestions
leveraging field and variable types.
Show meta informations on ES|QL query hover on policy names.
Show function signature on ES|QL query hover on function text.


### Checklist

Delete any items that are not applicable to this PR.

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [x] Any UI touched in this PR is usable by keyboard only (learn more
about [keyboard accessibility](https://webaim.org/techniques/keyboard/))

---------

Co-authored-by: kibanamachine <[email protected]>
Co-authored-by: Stratoula Kalafateli <[email protected]>
Co-authored-by: Abdon Pijpelink <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stratoula stratoula stratoula approved these changes

Assignees
No one assigned
Labels
Feature:ES|QL ES|QL related features in Kibana release_note:skip Skip the PR/issue when compiling release notes Team:Visualizations Visualization editors, elastic-charts and infrastructure v8.12.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@dej611 @elasticmachine @kibana-ci @stratoula