-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into improve-synth-alerting
- Loading branch information
Showing
63 changed files
with
2,033 additions
and
927 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,89 +1,38 @@ | ||
| # ES|QL utility library | ||
| # ES|QL AST library | ||
|
|
||
| ## Folder structure | ||
| The general idea of this package is to provide low-level ES|QL parsing, | ||
| building, traversal, pretty-printing, and manipulation features on top of a | ||
| custom compact AST representation, which is designed to be resilient to many | ||
| grammar changes. | ||
|
|
||
| This library brings all the foundation data structure to enable all advanced features within an editor for ES|QL as validation, autocomplete, hover, etc... | ||
| The package is structure as follow: | ||
| Contents of this package: | ||
|
|
||
| ``` | ||
| src | ||
| |- antlr // => contains the ES|QL grammar files and various compilation assets | ||
| | ast_factory.ts // => binding to the Antlr that generates the AST data structure | ||
| | ast_errors.ts // => error translation utility from raw Antlr to something understandable (somewhat) | ||
| | antlr_error_listener.ts // => The ES|QL syntax error listener | ||
| | antlr_facade.ts // => getParser and getLexer utilities | ||
| | ... // => miscellaneas utilities to work with AST | ||
| ``` | ||
|
|
||
| ### Basic usage | ||
|
|
||
| #### Get AST from a query string | ||
| - [`builder` — Contains the `Builder` class for AST node construction](./src/builder/README.md). | ||
| - [`parser` — Contains text to ES|QL AST parsing code](./src/parser/README.md). | ||
| - [`walker` — Contains the ES|QL AST `Walker` utility](./src/walker/README.md). | ||
| - [`visitor` — Contains the ES|QL AST `Visitor` utility](./src/visitor/README.md). | ||
| - [`pretty_print` — Contains code for formatting AST to text](./src/pretty_print/README.md). | ||
|
|
||
| This module contains the entire logic to translate from a query string into the AST data structure. | ||
| The `getAstAndSyntaxErrors` function returns the AST data structure, unless a syntax error happens in which case the `errors` array gets populated with a Syntax error. | ||
|
|
||
| ##### Usage | ||
| ## Demo | ||
|
|
||
| ```js | ||
| import { getAstAndSyntaxErrors } from '@kbn/esql-ast'; | ||
| Much of the functionality of this package is demonstrated in the demo UI. You | ||
| can run it in Storybook, using the following command: | ||
|
|
||
| const queryString = "from index | stats 1 + avg(myColumn) "; | ||
| const { ast, errors} = await astProvider(queryString); | ||
|
|
||
| if(errors){ | ||
| console.log({ syntaxErrors: errors }); | ||
| } | ||
| // do stuff with the ast | ||
| ```bash | ||
| yarn storybook esql_ast_inspector | ||
| ``` | ||
|
|
||
| ## How does it work | ||
|
|
||
| The general idea of this package is to provide all ES|QL features on top of a custom compact AST definition (all data structure types defined in `./types.ts`) which is designed to be resilient to many grammar changes. | ||
| The pipeline is the following: | ||
| Alternatively, you can start Kibana with *Example Plugins* enabled, using: | ||
|
|
||
| ```bash | ||
| yarn start --run-examples | ||
| ``` | ||
| Antlr grammar files | ||
| => Compiled grammar files (.ts assets in the antlr folder) | ||
| => AST Factory (Antlr Parser tree => custom AST) | ||
| ``` | ||
|
|
||
| Each feature function works with the combination of the AST and the definition files: the former describe the current statement in a easy to traverse way, while the definitions describe what's the expected behaviour of each node in the AST node (i.e. what arguments should it accept? How many arguments? etc...). | ||
| While AST requires the grammar to be compiled to be updated, definitions are static files which can be dynamically updated without running the ANTLR compile task. | ||
|
|
||
| #### AST | ||
|
|
||
| The AST is generated by 2 files: `ast_factory.ts` and its buddy `ast_walker.ts`: | ||
| * `ast_factory.ts` is a binding to Antlr and access the Parser tree | ||
| * Parser tree is passed over to `ast_walker` to append new AST nodes | ||
|
|
||
| In general Antlr is resilient to grammar errors, in the sense that it can produe a Parser tree up to the point of the error, then stops. This is useful to perform partial tasks even with broken queries and this means that a partial AST can be produced even with an invalid query. | ||
|
|
||
| ### Keeping ES|QL up to date | ||
|
|
||
| In general when operating on changes here use the `yarn kbn watch` in a terminal window to make sure changes are correctly compiled. | ||
|
|
||
| ### How to add new commands/options | ||
|
|
||
| When a new command/option is added to ES|QL it is done via a grammar update. | ||
| Therefore adding them requires a two step phase: | ||
| * Update the grammar with the new one | ||
| * add/fix all AST generator bindings in case of new/changed TOKENS in the `lexer` grammar file | ||
| * Update the definition files for commands/options | ||
| Then navigate to the *ES|QL AST Inspector* plugin in the Kibana UI. | ||
|
|
||
| To update the grammar: | ||
| 1. Make sure the `lexer` and `parser` files are up to date with their ES counterparts | ||
| * an existing Kibana CI job is updating them already automatically | ||
| 2. Run the script into the `package.json` to compile the ES|QL grammar. | ||
| 3. open the `ast_factory.ts` file and add a new `exit<Command/Option>` method | ||
| 4. write some code in the `ast_walker/ts` to translate the Antlr Parser tree into the custom AST (there are already few utilites for that, but sometimes it is required to write some more code if the `parser` introduced a new flow) | ||
| * pro tip: use the `http://lab.antlr.org/` to visualize/debug the parser tree for a given statement (copy and paste the grammar files there) | ||
| 5. if something goes wrong with new quoted/unquoted identifier token, open the `ast_helpers.ts` and check the ids of the new tokens in the `getQuotedText` and `getUnquotedText` functions - please make sure to leave a comment on the token name | ||
|
|
||
| #### Debug and fix grammar changes (tokens, etc...) | ||
| ## Keeping ES|QL AST library up to date | ||
|
|
||
| On TOKEN renaming or with subtle `lexer` grammar changes it can happens that test breaks, this can be happen for two main issues: | ||
| * A TOKEN name changed so the `ast_walker.ts` doesn't find it any more. Go there and rename the TOKEN name. | ||
| * TOKEN order changed and tests started failing. This probably generated some TOKEN id reorder and there are two functions in `ast_helpers.ts` who rely on hardcoded ids: `getQuotedText` and `getUnquotedText`. | ||
| * Note that the `getQuotedText` and `getUnquotedText` are automatically updated on grammar changes detected by the Kibana CI sync job. | ||
| * to fix this just look at the commented tokens and update the ids. If a new token add it and leave a comment to point to the new token name. | ||
| * This choice was made to reduce the bundle size, as importing the `esql_parser` adds some hundreds of Kbs to the bundle otherwise. | ||
| In general when operating on changes here use the `yarn kbn watch` in a terminal | ||
| window to make sure changes are correctly compiled. |
Oops, something went wrong.