[ES|QL] Batch of fixes and new features for autocomplete and validation #175986

dej611 · 2024-01-31T14:18:58Z

Summary

This PR is a batch of fixes and (tiny) new features for validation and autocomplete items in ES|QL client side area.
Content of the PR:

🐛 remove array/multi-value definition annotation for mv_* functions - fix [ES|QL] Use non array types for multi-value functions #174709
- while the type check has been relaxed for lists, the actual non-multivalued argument type is retained and used for validation
🐛 Remove array/multi-value definition annotation for mv_expand command - fix [ES|QL] Use non array types for multi-value functions #174709
✨ add metadata fields validation + autocomplete (fields are retrieved via Kibana uiSettings) ( part of [ES|QL] New grammar follow up tasks #172646 )

* ✨ Quick fixes now available also for metadata fields
🐛 fixed autocomplete for NOT operation
fixed autocomplete for in operation, with or without not:
- ✨ now a missing list bracket is correctly detected and suggested after in
- ✨ the content for the list is suggested and takes into account both the left side of the in/not in operator, but also the current content of the list

fixed grok and dissect in various ways
*:sparkles: the pattern provided by autocomplete for both was not valid in dissect nor useful at all, so I've changed to something more useful like '"%{WORD:firstWord}"' for grok and '"%{firstWord}"' for dissect to match the first word in the text field.
- 🐛 there was a bug in the validation engine as both grok and dissect could generate new columns based on matches, so now a new field query is fired (only when either a GROK or a DISSECT command is detected) which enriches the set of fields available (similar to the enrich fields)
- ✅ Added tons of tests here
🐛 fixed an issue with proposing an assign (=) operator when it should not
- ... EVAL var0 = round( ...) <here> not within another assignment
- ... EVAL var0 = 1 <here> not within another assignment
- ... EVAL var0 = 1 year + 2 <here> not within another assignment
- ... | WHERE field should not shadow a field name in this case
🐛 fix an annoying auto trigger suggestion on field selection, and kept it only for functions
🐛 fixed an error in console with autocomplete when a typed function does not exists
✅ Add tests for the hover feature
🔥 Removed some unused functions detected via code coverage analysis

Checklist

Unit or functional tests were updated or added to match the most common scenarios

dej611 · 2024-01-31T14:20:31Z

/ci

dej611 · 2024-02-01T10:46:57Z

/ci

elasticmachine · 2024-02-01T10:49:15Z

Pinging @elastic/kibana-visualizations (Team:Visualizations)

stratoula · 2024-02-01T11:04:59Z

/ci

dej611 · 2024-02-01T11:34:12Z

/ci

…bana into fix/esql-various-fixes

dej611 · 2024-02-01T11:37:50Z

/ci

stratoula

Love all the fixes, I am just not so sure about the metadata validation and autocomplete.

stratoula · 2024-02-01T11:55:01Z

packages/kbn-text-based-editor/src/text_based_languages_editor.tsx

@@ -341,6 +342,7 @@ export const TextBasedLanguagesEditor = memo(function TextBasedLanguagesEditor({
        }
        return [];
      },
+      getMetaFields: async () => uiSettings?.get('metaFields') || [],


Why do we get them from the advanced setting? I think is better to hardcode them. Check below. I have removed the _id but if I use it in my query it doesn't fail. So we should not error out or not suggest it imho

Who decides what's a metafield or not?
Right now this information can be configured by Users/Admins into Kibana Settings, and we're using them already.
Let me ask the ES team where are they sourcing metadata fields.

I get your point but I dont think that is correct to error out a query which is not wrong and is going to return results. I am pretty sure that there is a list we can use without retrieving this information from the ui setting.

Here it is https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-fields.html

I've found in the ES source code a hardcoded list of metadata fields:

_version

_index

_source

_id

Discussed offline with the ES ES|QL team and they suggested to adopt their own hardcoded list of fields.
Changed here 6faf762

packages/kbn-text-based-editor/src/text_based_languages_editor.tsx

….tsx

dej611 · 2024-02-01T14:24:45Z

/ci

stratoula

Very nice enhancements Marco! LGTM! 💃

kibana-ci · 2024-02-01T15:36:56Z

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/monaco`	104	105	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`textBasedLanguages`	154.1KB	154.2KB	+61.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`kbnUiSharedDeps-srcJs`	2.9MB	2.9MB	+3.0KB

Unknown metric groups

API count

id	before	after	diff
`@kbn/monaco`	104	105	+1

History

💔 Build #190766 failed 6faf762
💚 Build #190725 succeeded 41ddfd6
💔 Build #190712 failed 8af04fb
💔 Build #190711 failed 8af04fb
💔 Build #190701 failed df467ed
💔 Build #190680 failed 9a5d10d

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…on (elastic#175986) ## Summary This PR is a batch of fixes and (tiny) new features for validation and autocomplete items in ES|QL client side area. Content of the PR: * 🐛 remove array/multi-value definition annotation for `mv_*` functions - fix elastic#174709 <img width="305" alt="Screenshot 2024-01-31 at 13 43 37" src="https://github.com/elastic/kibana/assets/924948/2a458e74-9c87-4a67-8b42-0071e7a04f48"> <img width="287" alt="Screenshot 2024-01-31 at 13 43 25" src="https://github.com/elastic/kibana/assets/924948/7205d984-d0c9-410b-8b39-c1d530c3f54f"> * while the type check has been relaxed for lists, the actual non-multivalued argument type is retained and used for validation <img width="758" alt="Screenshot 2024-01-31 at 13 44 06" src="https://github.com/elastic/kibana/assets/924948/384e9dfa-aaf8-4e0f-953e-a2d656776ea7"> * 🐛 Remove array/multi-value definition annotation for `mv_expand` command - fix elastic#174709 <img width="556" alt="Screenshot 2024-01-31 at 14 36 19" src="https://github.com/elastic/kibana/assets/924948/0debbccc-da9a-4e6a-bfc9-3fded61300b5"> <img width="277" alt="Screenshot 2024-01-31 at 14 36 13" src="https://github.com/elastic/kibana/assets/924948/9438832b-d8d5-48c5-a030-b8983057546e"> * ✨ add metadata fields validation + autocomplete (fields are retrieved via Kibana `uiSettings`) ( part of elastic#172646 ) ![metadata_fix](https://github.com/elastic/kibana/assets/924948/958e6609-ab73-4949-8652-f368c6604a09) <img width="760" alt="Screenshot 2024-01-31 at 13 45 43" src="https://github.com/elastic/kibana/assets/924948/b3f13d99-3e12-4586-b39c-f4ea3691a2f1"> * ✨ Quick fixes now available also for metadata fields ![metadata_fixes](https://github.com/elastic/kibana/assets/924948/b273418f-a754-4c50-9539-a128f35fc4cd) * 🐛 fixed autocomplete for `NOT` operation <img width="549" alt="Screenshot 2024-01-31 at 13 46 43" src="https://github.com/elastic/kibana/assets/924948/29b7aa7c-db66-485a-a5b0-329e66df842f"> * fixed autocomplete for `in` operation, with or without `not`: * ✨ now a missing list bracket is correctly detected and suggested after `in` * ✨ the content for the list is suggested and takes into account both the left side of the `in`/`not in` operator, but also the current content of the list ![in_list_fix_2](https://github.com/elastic/kibana/assets/924948/50ea8db5-23da-411c-b81e-5ba3397b65c1) ![in_list_fix](https://github.com/elastic/kibana/assets/924948/cbb73ad2-4073-4eb6-9c12-61835858fd5e) ![not_in_list_fix](https://github.com/elastic/kibana/assets/924948/9d7fa756-3d34-49fe-ac7a-28e2edbe7fee) * fixed `grok` and `dissect` in various ways *:sparkles: the pattern provided by autocomplete for both was not valid in `dissect` nor useful at all, so I've changed to something more useful like `'"%{WORD:firstWord}"'` for `grok` and `'"%{firstWord}"'` for `dissect` to match the first word in the text field. * :bug: there was a bug in the validation engine as both `grok` and `dissect` could generate new columns based on matches, so now a new field query is fired (only when either a `GROK` or a `DISSECT` command is detected) which enriches the set of fields available (similar to the enrich fields) * ✅ Added tons of tests here ![grok_fix](https://github.com/elastic/kibana/assets/924948/a23c8edc-0702-4531-aaa5-6126e375913b) ![dissect_fix](https://github.com/elastic/kibana/assets/924948/d5591753-775e-4768-be00-31c4371be330) * 🐛 fixed an issue with proposing an assign (`=`) operator when it should not * `... EVAL var0 = round( ...) <here>` not within another assignment * `... EVAL var0 = 1 <here>` not within another assignment * `... EVAL var0 = 1 year + 2 <here>` not within another assignment * `... | WHERE field` should not shadow a field name in this case * 🐛 fix an annoying auto trigger suggestion on field selection, and kept it only for functions ![auto_trigger_fix](https://github.com/elastic/kibana/assets/924948/31d0c296-6338-450d-8f81-e08cd1c7526d) * 🐛 fixed an error in console with autocomplete when a typed function does not exists * ✅ Add tests for the hover feature * 🔥 Removed some unused functions detected via code coverage analysis ### Checklist - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: Stratoula Kalafateli <[email protected]>

dej611 added 23 commits January 30, 2024 15:39

🔧 export utility fn

b90808c

🔧 Change to non-array type

78c4ec3

🔥 Remove unused fn

8665d2c

🔥 Remove unused branches and fns

eb5697d

🏷️ Add types

1db565f

✨ Enable metafields

bc1e204

🐛 Fix issues with not and in oeprators

4fa664e

✨ Add metadata validation

9838f47

✨ Add metadata fields callback

1b8a3a0

✨ Tons of fixes

3a1819c

✨ Add autocomplete support/fixes

0d920d4

✨ Add validation support/fixes

623caa9

Merge remote-tracking branch 'upstream/main' into fix/esql-various-fixes

8b50c59

🐛 Fix for not in expressions

98d89fb

✅ Add tests for hover

2446122

🚚 rename file

abffd85

🐛 better detect newExpression

9b44170

✅ Add more tests

fa9b683

🐛 Suggest a better pattern

68784b3

✨ Use a better pattern for grok and dissect

ccd0687

🐛 Fix missing column on grok/dissect match

62b3298

🐛 Avoid auto trigger on fields

33d2083

🔥 Removed debug logging

6242474

dej611 added Team:Visualizations Visualization editors, elastic-charts and infrastructure release_note:skip Skip the PR/issue when compiling release notes Feature:ES|QL ES|QL related features in Kibana v8.13.0 labels Jan 31, 2024

Merge branch 'main' into fix/esql-various-fixes

aae41c7

Merge branch 'main' into fix/esql-various-fixes

8d1abc9

✨ Extends auto fixes to metadata fields

df467ed

dej611 marked this pull request as ready for review February 1, 2024 10:49

dej611 requested a review from a team as a code owner February 1, 2024 10:49

Merge branch 'main' into fix/esql-various-fixes

8af04fb

🐛 fix import

3e38d82

Merge branch 'fix/esql-various-fixes' of https://github.com/dej611/ki…

41ddfd6

…bana into fix/esql-various-fixes

stratoula reviewed Feb 1, 2024

View reviewed changes

dej611 commented Feb 1, 2024

View reviewed changes

packages/kbn-text-based-editor/src/text_based_languages_editor.tsx Outdated Show resolved Hide resolved

dej611 commented Feb 1, 2024

View reviewed changes

packages/kbn-text-based-editor/src/text_based_languages_editor.tsx Outdated Show resolved Hide resolved

♻️ Revisit the metadata fields

6faf762

dej611 commented Feb 1, 2024

View reviewed changes

packages/kbn-text-based-editor/src/text_based_languages_editor.tsx Outdated Show resolved Hide resolved

dej611 added 2 commits February 1, 2024 15:11

Update packages/kbn-text-based-editor/src/text_based_languages_editor…

9a95bdf

….tsx

Merge branch 'main' into fix/esql-various-fixes

07d1c80

stratoula approved these changes Feb 1, 2024

View reviewed changes

dej611 enabled auto-merge (squash) February 1, 2024 14:37

mistic disabled auto-merge February 1, 2024 16:18

mistic merged commit af9951f into elastic:main Feb 1, 2024
18 of 19 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Feb 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ES|QL] Batch of fixes and new features for autocomplete and validation #175986

[ES|QL] Batch of fixes and new features for autocomplete and validation #175986

dej611 commented Jan 31, 2024 •

edited

Loading

dej611 commented Jan 31, 2024

dej611 commented Feb 1, 2024

elasticmachine commented Feb 1, 2024

stratoula commented Feb 1, 2024

dej611 commented Feb 1, 2024

dej611 commented Feb 1, 2024

stratoula left a comment

stratoula Feb 1, 2024

dej611 Feb 1, 2024

stratoula Feb 1, 2024

stratoula Feb 1, 2024

dej611 Feb 1, 2024

dej611 Feb 1, 2024

dej611 commented Feb 1, 2024

stratoula left a comment

kibana-ci commented Feb 1, 2024

API count

[ES|QL] Batch of fixes and new features for autocomplete and validation #175986

[ES|QL] Batch of fixes and new features for autocomplete and validation #175986

Conversation

dej611 commented Jan 31, 2024 • edited Loading

Summary

Checklist

dej611 commented Jan 31, 2024

dej611 commented Feb 1, 2024

elasticmachine commented Feb 1, 2024

stratoula commented Feb 1, 2024

dej611 commented Feb 1, 2024

dej611 commented Feb 1, 2024

stratoula left a comment

Choose a reason for hiding this comment

stratoula Feb 1, 2024

Choose a reason for hiding this comment

dej611 Feb 1, 2024

Choose a reason for hiding this comment

stratoula Feb 1, 2024

Choose a reason for hiding this comment

stratoula Feb 1, 2024

Choose a reason for hiding this comment

dej611 Feb 1, 2024

Choose a reason for hiding this comment

dej611 Feb 1, 2024

Choose a reason for hiding this comment

dej611 commented Feb 1, 2024

stratoula left a comment

Choose a reason for hiding this comment

kibana-ci commented Feb 1, 2024

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

Page load bundle

API count

History

dej611 commented Jan 31, 2024 •

edited

Loading